View previous topic :: View next topic |
Author |
Message |
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3637 Location: Rasi, Finland
|
Posted: Sun Feb 19, 2017 6:01 pm Post subject: InfiniBand - a cheap way to _fast_ network, PC to PC? |
|
|
I stumbled into this youtube video.
Then I went straight to eBay and bought cheapest 20Gbps IB cards and a cable. Reason? I need faster transfers between my desktop and server for backups copying raw video etc.
Time will tell if I just threw close to 100€ into a well...
Meanwhile I'd like to hear if any of you have experience with InfiniBand and if you use it with Gentoo specifically.
Last edited by Zucca on Wed Feb 22, 2017 7:22 pm; edited 1 time in total |
|
Back to top |
|
|
R0b0t1 Apprentice
Joined: 05 Jun 2008 Posts: 264
|
Posted: Wed Feb 22, 2017 5:16 am Post subject: |
|
|
I've been looking for something similar. Has it worked?
InfiniBand seems like a cheaper, better, faster option than a local highspeed copper (of some other technology) or fiber network. I suppose supercomputing consumers may have driven the price down - it seems too cheap for what it is compared to the alternatives. |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3637 Location: Rasi, Finland
|
Posted: Wed Feb 22, 2017 6:14 pm Post subject: |
|
|
R0b0t1 wrote: | I've been looking for something similar. Has it worked? | The cards have not arrived yet.
R0b0t1 wrote: | than a local highspeed copper (of some other technology) or fiber network. | InfiniBand can also be copper or optical.
But yeah. It seems quite cheap. I thought of going to 40Gbps, but I suspect I couldn't really utilize that fast connection.
I ordered the 4x cards and a cable. There is also a 12x connector. Which is quite wide.
One thing I haven't figured out completely yet is why there's RAM memory on some cards. Maybe if you directly attach a stogare device(s) to the cable the it can act as a fast cache maybe... _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
John R. Graham Administrator
Joined: 08 Mar 2005 Posts: 10652 Location: Somewhere over Atlanta, Georgia
|
Posted: Wed Feb 22, 2017 6:24 pm Post subject: |
|
|
Ooohh. I'm interested in the outcome of this experiment, too. I have a baby cluster at home that I would love to network with InfiniBand.
Edit: I don't see a combination that at 4x links produces 20Gb/sec throughput. What type of cards did you order?
- John _________________ I can confirm that I have received between 0 and 499 National Security Letters. |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3637 Location: Rasi, Finland
|
Posted: Wed Feb 22, 2017 7:10 pm Post subject: |
|
|
John R. Graham wrote: | What type of cards did you order? | Mellanox MHGS18-XTC. Oh, and look at the price (at introduction, I guess). ;P I paid around $30-$35 for each.
Note that I'm using bps as "bits per second". _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
John R. Graham Administrator
Joined: 08 Mar 2005 Posts: 10652 Location: Somewhere over Atlanta, Georgia
|
Posted: Wed Feb 22, 2017 7:59 pm Post subject: |
|
|
Got it. I think that rate is without encoding overhead. The actual data throughput is 16 Gbit/sec. Come to think of it, though, Gigabit Ethernet probably advertises raw line rate, too, so this is probably fully 20x faster than GigE.
- John _________________ I can confirm that I have received between 0 and 499 National Security Letters. |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3637 Location: Rasi, Finland
|
Posted: Wed Feb 22, 2017 8:10 pm Post subject: |
|
|
John R. Graham wrote: | so this is probably fully 20x faster than GigE. | I do hope so. |
|
Back to top |
|
|
Ant P. Watchman
Joined: 18 Apr 2009 Posts: 6920
|
Posted: Wed Feb 22, 2017 9:35 pm Post subject: |
|
|
Since it seems relevant to the topic, has anyone ever tried IP over Firewire? Seems like a cheap way to get up to 3.2Gbps uninterrupted bandwidth between two PCs if they both have FW ports sitting otherwise idle; I just need a second one to play with... |
|
Back to top |
|
|
R0b0t1 Apprentice
Joined: 05 Jun 2008 Posts: 264
|
Posted: Wed Feb 22, 2017 9:52 pm Post subject: |
|
|
Zucca wrote: | One thing I haven't figured out completely yet is why there's RAM memory on some cards. Maybe if you directly attach a stogare device(s) to the cable the it can act as a fast cache maybe... |
Unfortunately I don't have a very in-depth answer for you, but suffice it to say that routing logic isn't stateless. Some of the RAM may be for buffering received data but it may also be used for configuring the behavior of the network interface (i.e. acting on the raw internet protocol and underlying physical interface data). As InfiniBand is not as widely implemented as the various forms of ethernet they may not have miniaturized the hardware to the point where then necessary memory is on-die with the controlling logic.
Probably a fruitless endeavor, but the cards may have a miniature RTOS that you could run code on if you could figure out how to access it.
Thanks for info w.r.t. what you ordered - I'll give it a look when time and money permit.
Last edited by R0b0t1 on Thu Feb 23, 2017 5:24 pm; edited 1 time in total |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3637 Location: Rasi, Finland
|
Posted: Thu Feb 23, 2017 10:53 am Post subject: |
|
|
Ant P. wrote: | Since it seems relevant to the topic, has anyone ever tried IP over Firewire? Seems like a cheap way to get up to 3.2Gbps uninterrupted bandwidth between two PCs if they both have FW ports sitting otherwise idle; I just need a second one to play with... | Isn't the FW3200 made mainly for server environments? I recall that FW800 was the best consumer could "get".
But yeah. FW3200 would yield to quite high speeds too. I haven't tried. I think FW has DMA. I think that drops latency quite a lot, making it proper for certain networking tasks. _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3637 Location: Rasi, Finland
|
Posted: Fri Feb 24, 2017 11:37 am Post subject: |
|
|
I have small backfiring on my planned setup.
The PCIe 16x slot on my desktop motherboard only acts as 4x, while the card needs at least 8x. I read that with 4x I can (could) still get around 16Gbps, so I'll settle with that until I upgrade to (possibly) Ryzen.
In more distant future I might upgrade to Thunderbolt networking. _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
John R. Graham Administrator
Joined: 08 Mar 2005 Posts: 10652 Location: Somewhere over Atlanta, Georgia
|
Posted: Fri Feb 24, 2017 11:59 am Post subject: |
|
|
If your PCIe slots are v2.0, then that's correct, but it's the theoretical maximum. In practice, I'd expect half that, which would still be enough to declare your experiment a resounding success, in my opinion.
- John _________________ I can confirm that I have received between 0 and 499 National Security Letters. |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3637 Location: Rasi, Finland
|
Posted: Fri Feb 24, 2017 12:18 pm Post subject: |
|
|
John R. Graham wrote: | If your PCIe slots are v2.0, then that's correct, but it's the theoretical maximum. In practice, I'd expect half that, which would still be enough to declare your experiment a resounding success, in my opinion.
- John | Yes. PCIe 2.0. A half of 16Gbps sounds a bit pessimistic, but I can cope with it.
Wikipedia wrote: | Like 1.x, PCIe 2.0 uses an 8b/10b encoding scheme, therefore delivering, per-lane, an effective 4 Gbit/s max transfer rate from its 5 GT/s raw data rate. | ... does GT mean GigaTransfers? Is there some error correcting bits (checksums) so that theoretical speed is actually 4/5th of the speed?
I'm not too familiar with the inner side of PCIe technology... _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
John R. Graham Administrator
Joined: 08 Mar 2005 Posts: 10652 Location: Somewhere over Atlanta, Georgia
|
Posted: Fri Feb 24, 2017 3:09 pm Post subject: |
|
|
Zucca wrote: | ... does GT mean GigaTransfers? Is there some error correcting bits (checksums) so that theoretical speed is actually 4/5th of the speed?
I'm not too familiar with the inner side of PCIe technology... | Yes, GT means that. For the rest, during the design of any signaling medium, the designers have to answer two questions:- How do you distinguish the boundary between one symbol and the next?
- Since it's desirable to be able to transmit all possible symbols in a message, what mechanism is used to distinguish between actual message and metadata (start of message, length of message, checksum info, etc.)?
Of course there are many other questions to be answered, but these two will suffice to explain the 4/5 overhead you've read about. In very general terms, there are two ways to distinguish those things:- With out-of-band signaling: using extra wires (or radio channels, or carrier pigeons) to delineate the boundaries between symbols and message elements.
- With in-band signaling: using the same wires (or radio channels, or carrier pigeons) with special signal patterns to delineate such things. Those special patterns eat up a portion of the bandwidth that could otherwise be used to move data, but then the physical interface is simpler, using less wires (or...well, you get the idea).
Both question 1 and question 2 can independently be answered with answer 1 or answer 2, by the way.
- John _________________ I can confirm that I have received between 0 and 499 National Security Letters. |
|
Back to top |
|
|
1clue Advocate
Joined: 05 Feb 2006 Posts: 2569
|
Posted: Fri Feb 24, 2017 5:06 pm Post subject: |
|
|
I'm interested too, although probably more at a 10gbps rate. Still, IB sounds interesting and as yet I have found precious few Linux posts about it. |
|
Back to top |
|
|
szatox Advocate
Joined: 27 Aug 2013 Posts: 3348
|
Posted: Fri Feb 24, 2017 8:18 pm Post subject: |
|
|
I'm watching this topic too.
I wonder how you managed to get those adapters at 35$ each. When you mentioned this price I started looking around, but the things I found tend to be MUCH more expensive. Prohibitively expensive.
My more technical questions are:
Can you use IB as a boot medium? I mean something like PXE or SAN boot.
Can you use its RDMA feature for high performance storage sharing? (Stretched RAID anyone?) |
|
Back to top |
|
|
R0b0t1 Apprentice
Joined: 05 Jun 2008 Posts: 264
|
Posted: Fri Feb 24, 2017 8:38 pm Post subject: |
|
|
szatox wrote: | I'm watching this topic too.
I wonder how you managed to get those adapters at 35$ each. When you mentioned this price I started looking around, but the things I found tend to be MUCH more expensive. Prohibitively expensive.
My more technical questions are:
Can you use IB as a boot medium? I mean something like PXE or SAN boot.
Can you use its RDMA feature for high performance storage sharing? (Stretched RAID anyone?) |
For the first: You should be able to, but the UEFI firmware needs a driver. If the IB cards support a common subset of communications commands, similar to graphics cards for their display capabilities, it should work. I've seen netbooting ARM boards over USB to Ethernet converters but this was something with explicit support in the bootloader.
Very interested in #2. |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3637 Location: Rasi, Finland
|
Posted: Fri Feb 24, 2017 10:38 pm Post subject: |
|
|
szatox wrote: | I wonder how you managed to get those adapters at 35$ each. When you mentioned this price I started looking around, but the things I found tend to be MUCH more expensive. Prohibitively expensive. |
Also I think you could find used IB stuff from Amazon as well...
Also just in case... I'll mention this:
Be sure to pick ones with PCIe, not PCI-X... unless, of course, you happen to have PCI-X slots on your MB. _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3637 Location: Rasi, Finland
|
Posted: Sat Feb 25, 2017 11:09 am Post subject: |
|
|
szatox wrote: | Can you use its RDMA feature for high performance storage sharing? (Stretched RAID anyone?) | There are IB-SATA cables. I guess you have to specifically configure the IB port to act as SATA host. I don't know if the IB host card needs some specific hardware feature to be able to act as so.
Earlier I found a used IB switch from ebay for about $100. I think it was 10Gbps one. Just imagine how many SATA drives you could attach... If it's even possible that way.
I really wish there was someone with actual experience with using IB on large servers.
And thanks John for the insights of PCIe! |
|
Back to top |
|
|
unixlike n00b
Joined: 18 Dec 2016 Posts: 4
|
Posted: Mon Feb 27, 2017 9:09 am Post subject: |
|
|
I've seen such cards on ebay myself since there were always posts about it as a solution on hardforum five years ago, on gentoo forums two years ago.. But I didn't buy because of the drivers. Aren't those cards gonna use the mthca driver as opposed to the newer (world apart cards) that use mlx5? |
|
Back to top |
|
|
szatox Advocate
Joined: 27 Aug 2013 Posts: 3348
|
Posted: Mon Feb 27, 2017 7:05 pm Post subject: |
|
|
Well, it's getting more interesting.
Talking about RDMA, I didn't mean IB-SATA wires, I've been thinking about computer-computer connection where one machine uses hard drives owned by the other. Pretty much the same way it would work with iSCSI, but skipping all the CPU-related overhead. |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3637 Location: Rasi, Finland
|
Posted: Tue Feb 28, 2017 11:25 am Post subject: |
|
|
szatox wrote: | but skipping all the CPU-related overhead. | I'm currently looking for options for that. Haven't decided yet.
This may be something to consider... If it's possible. _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3637 Location: Rasi, Finland
|
Posted: Tue Feb 28, 2017 12:24 pm Post subject: |
|
|
Apparently the post man was on sick leave.
The post office has a ninja in place of him.
Came to the gates and took an U-turn so fast that nobody saw him. Not even the dogs outside.
I think that's a record.
So I'll be picking up the cards myself from the local post office then. _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
szatox Advocate
Joined: 27 Aug 2013 Posts: 3348
|
Posted: Sat Mar 18, 2017 11:52 am Post subject: |
|
|
So.... did you manage to get anything usable out of it? |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3637 Location: Rasi, Finland
|
Posted: Sun Mar 19, 2017 2:29 pm Post subject: |
|
|
I went and bought some other parts too. I decided to try out one thing I've always thought of trying out (You'll propably see that later).
I have all but one part now. I could, however, plug in the IB cards today already. I would need to maybe rebuild the kernel for the new motherboard and switch that motherboard in before I could actually test those IB cards.
The last part should arrive next week, so for the conveniece I'll pull all the things together at the same time.
Sorry for the delay. I'm very impatiently waiting to see the results. _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
|