Hey,

I have a scenario which I have a server with static IP that I want to use to access the RouterOS remote devices and to allow access to the devices from couple static ip addresses.
To allow other users to access the remote devices I want to use a http proxy which winbox should use.

Is it possible to somehow connect winbox via a HTTP proxy?

Winbox is not using HTTP or HTTPS. It is using TCP port 8291 for it's connections. HTTP proxy does not help here.
Webfig does HTTP/HTTPS, and has a similar set of GUI settings. So here proxy is relevant.

Proxy for WinBox is rather RoMon , a different L2 protocol MAC address based, not L3 (IP address) based.
Connection to RoMon from WinBox is also possible in L3 (IP route based)
Telnet (and SSH) can be done with ROS as intermediate point.

Otherwise (VPN) tunnels are the way to have network access to remote devices, via a central point.
And why not, even port forwarding (DST NAT) could be used.

OK, the VPN way seems like the right way.