An RPI4 based file server

[vtcnc]

I know that this is Hobby Machinist but I’m interested in hearing about the software. Reality is that members here will continue to adopt technology that helps them make more chips. As tools get smaller, stronger, faster and more available to the broader population, it only makes sense that this group learns the technology behind what operates them. Or in some cases how to build them.


Sent from my iPhone using Tapatalk

 

[WobblyHand]

@vtcnc which parts of the software? RAID, rsync, overclocking, boot from SSD, pwm fan, something else? Kind of a broad area to cover. Kind of like me asking "how to machine" here on HM. If there is something specific you would like to know, just ask. If I know the answer, I'll answer right away. If I don't know the answer, I'll try to find out. If I'm clueless, I'll say I'm stumped.

Most of this stuff, I've learned by doing, and occasionally screwing up. Say you trash the SD memory card, what are you out? Just the $5-$10 for another SD card. The entry price for an RPI is relatively inexpensive. The Raspberry Pi Forum and and raspberrypi.org make it pretty easy to get going, no matter which kind of computer you are starting from. Linux can be a little intimidating initially, but Raspberry Pi has made it less so with their nice GUI. If the RPI is running from a SSD, it is pretty decent. No, it isn't a fire breathing gamer, but it's ok for most use. It is definitely good enough for a browsing in a shop. An RPI4 can drive two monitors - I had my RPI4 set up as a primary computer for 2-1/2 months after my laptop of 7 years just died. (Motherboard power supply failure! Effectively not repairable.) I got by using the RPI4. Have a great deal of respect for this sub $100 computer.

My plan is to make another duplicate set of HW. The idea is to have the second set off-site in case something happens (like a fire). The second server will mirror the first server.

Will also make a third unit with lesser capabilities as a "shop computer". For the shop computer, I will probably enclose it better as one doesn't want any flying metal chips to interact with silicon chips! How to set up a shop computer in my incredibly small and disorganized space, well, haven't designed it yet, heck, I don't even have a concept yet.

 

[WobblyHand]

For what it is worth, for the fan controller I used the software found at HW PWM fan controller for RPI. It seems to be based on a Noctua fan. I didn't have one. Instead I had a far cheaper brushless DC motor 30mm x 30mm x 7mm thick, which came with the RPI4 case I purchased. The Noctua would work great, but requires a new top layer lamination for the case. The Noctua is 40mm x 40mm. I'll try machining a new top layer at a later time. Being a little lazy, I decided to use the existing fan. The Noctua can be directly driven by the RPI4 since it is a 3 wire device. My fan could not be driven that way since it requires 200mA and only has 2 wires. An RPI4 pin can source at most maybe 10mA. My circuit, as stated in an earlier post, was a 1K resistor, a MPS2222A transistor, and a diode. I tried both a 1N4001 and a 1N4148 diode. The 1N4148 diode is a little faster and snubs the inductive transient a bit better. Either works fine. Schematic below.

Used a 40 pin connector with 1" pigtails to be able to plug the connector into the RPI4 and still have 5mm of the pins protrude out the top, should there be need to access the GPIO pins.

The software uses the BCM2835 library and can control the PWM pins. The gist repository has the links to that library. All the above software has to be "built" on your computer. That means you need to have the means to build software. You can build it on the RPI4 using gcc and Make. Had to modify some of the PWM parameters so my fan would start. Primary change was to slow down the PWM frequency to something my poor fan could follow. I think the PWM frequency is now about 100 Hz, rather than the 25KHz that a Noctua uses.

If anyone has questions on getting the fan to work, just let me know.

 

[WobblyHand]

Since my concept is to have the server initiate the backups, rather than the clients, I will need to install some software on both the server and clients. I want the server to make backups in the background, rather than having the client initiate a backup. Why? Because most of us, myself included, are irresponsible, and fail to backup regularly. So to make this happen, I need to install ssh-server on my clients. And I need to install ssh-client on my server. Yes this is confusing nomenclature... But to ssh, my server is requesting data from the client, so my file-server is a ssh-client.

So on the client (my laptop) I install openssh-server. Apt suggests ncurses-term, openssh-server, open_sftp-server, and ssh-import-id be installed. I also installed sshguard, to protect my laptop from brute force ssh attacks.

On the file server (the RPI4) I install openssh-client. In my case, openssh-client is already there, so nothing is installed. What we need to do now is come up with a script we can run periodically to perform an rsync (remote sync) on the clients (laptops, things that need backing up) to the file server. The way to do this periodic script is to use cron on the RPI. We will get back to cron later,

Rsync is a pretty interesting and powerful program that will synchronize files. These files can be on a single disk, or multiple disks, or between computers. The first time rsync runs, it will effectively do a copy from SOURCE to DESTINATION. The second time rsync runs it will check to see if a SOURCE file (or files) is newer than the DESTINATION. If it is newer, the file will be updated on the DESTINATION.

When my laptop failed in January, I used rsync to copy a lot of files to my RPI4, so I could get by. One thing I learned right away was the -n or
--dry-run option. This is very important to use when one is "trying out" rsync. It's very easy to inadvertently do something wrong, which can be time consuming to undo. So add the -n option to a new rsync command BEFORE testing it for real. The dry run option will print out what the command would have done. If you think it did what you expected, then just remove the "n" option and try again.

At the moment, I'm not sure what rsync options to use. The choices for me are -a (archive mode which is recursive and preserves all file characteristics like file date, access control levels, user, group privileges), or -b (backup mode). It seems that backup mode is more appropriate, but I need to make sure the file characteristics are preserved. I'll have to try it. If I specify --backup-dir=DIR, then rsync will place that backup in the directory named DIR. I may do this and have 7 days of backup. When the 7 days are up, it will write over the week old data. Now I have to go off and think about how to do this. And do a dry run!

In linux, there are man files. So to get basic information on a utility or command type $ man "command". You will be rewarded (or punished, depending on your perspective) with a lot of detailed information on the command. The man file for rsync is 2822 lines. There's a ton of information there. Too much. Off to research some details on rsync... What I do want to do is rsync via ssh - that way the data is encrypted. It may not matter on a home network, but if one eventually wants to backup over a public network, or have a remote server, using ssh is essential.

 

[WobblyHand]

Foo, have some sort of hardware error. One of the RAID1 disks is not online. Might be due to my clamping of the metal NVME cases? When I took it apart and had it out of the case it worked. I put some kapton tape over the nvme stick, but maybe there is some sort of intermittent connection. These kind of things are hard to find...

To tell the status of your raid array type: $ sudo mdadm --detail /dev/mdxx, where mdxx is your RAID device. For me, the device is md0.
In my case it says: State: clean, degraded
Active Devices: 1
Working Devices: 1
Disk0's state is "removed".

Have to take the assembly apart again. I'll fix it somehow. Might be the case, or the NVME needs to be reseated. Took it apart. Reassembled, but not as tight. Disk is online, but the RAID1 array isn't recognizing the disk. This isn't going to plan. I have some questions for my local Linux Users Group (LUG). Have to walk away from this for a bit, getting nowhere in a hurry. Tomorrow will be better!

 

[jwmelvin]

[mention]WobblyHand [/mention] does raspbian not support zfs? I find it much more enjoyable than mdadm.

 

[vtcnc]

@vtcnc which parts of the software? RAID, rsync, overclocking, boot from SSD, pwm fan, something else? Kind of a broad area to cover. Kind of like me asking "how to machine" here on HM. If there is something specific you would like to know, just ask. If I know the answer, I'll answer right away. If I don't know the answer, I'll try to find out. If I'm clueless, I'll say I'm stumped.

Most of this stuff, I've learned by doing, and occasionally screwing up. Say you trash the SD memory card, what are you out? Just the $5-$10 for another SD card. The entry price for an RPI is relatively inexpensive. The Raspberry Pi Forum and and raspberrypi.org make it pretty easy to get going, no matter which kind of computer you are starting from. Linux can be a little intimidating initially, but Raspberry Pi has made it less so with their nice GUI. If the RPI is running from a SSD, it is pretty decent. No, it isn't a fire breathing gamer, but it's ok for most use. It is definitely good enough for a browsing in a shop. An RPI4 can drive two monitors - I had my RPI4 set up as a primary computer for 2-1/2 months after my laptop of 7 years just died. (Motherboard power supply failure! Effectively not repairable.) I got by using the RPI4. Have a great deal of respect for this sub $100 computer.

My plan is to make another duplicate set of HW. The idea is to have the second set off-site in case something happens (like a fire). The second server will mirror the first server.

Will also make a third unit with lesser capabilities as a "shop computer". For the shop computer, I will probably enclose it better as one doesn't want any flying metal chips to interact with silicon chips! How to set up a shop computer in my incredibly small and disorganized space, well, haven't designed it yet, heck, I don't even have a concept yet.

I'm planning a build for a CNC lathe. So my motivation comes from needing to build a small shop computer that can run Centroid CNC on it. I'll have to go back and look at the specs for Centroid. However, I have a shop computer already for finding files on the net, Fusion 360 and of course frequenting here at H-M. I also use it for work when I need a meeting space and am working from home - which isn't all that often any more.

Anyway, I have a laptop in the house which I use primarily for Fusion 360 and file storage. It would be great if I could sync files between the shop computer, future lathe PC and the house laptop in the future. This project seems like it would fit the bill.

Sorry, I still have answered your question. It would be awesome if you could point to a git hub page or something that has all of the requisite software in one place. This is one of those situations where you don't know what you don't know - so my ask is not going to be really clear.

 

[WobblyHand]

[mention]WobblyHand [/mention] does raspbian not support zfs? I find it much more enjoyable than mdadm.

Hadn't considered zfs before. I've been reading about it, could be a viable solution for me, assuming I can fix my nvme device. Thanks for the idea. Not sure if zfs is on raspbian, but it is on debian, so there's a decent chance. My reading showed that it was originally designed for a 64 bit system. Right now my Raspberry Pi OS is 32 bits, (new name for Raspbian) as the 64 bit version is not quite ready.

 

[WobblyHand]

I'm planning a build for a CNC lathe. So my motivation comes from needing to build a small shop computer that can run Centroid CNC on it. I'll have to go back and look at the specs for Centroid. However, I have a shop computer already for finding files on the net, Fusion 360 and of course frequenting here at H-M. I also use it for work when I need a meeting space and am working from home - which isn't all that often any more.

Anyway, I have a laptop in the house which I use primarily for Fusion 360 and file storage. It would be great if I could sync files between the shop computer, future lathe PC and the house laptop in the future. This project seems like it would fit the bill.

Sorry, I still have answered your question. It would be awesome if you could point to a git hub page or something that has all of the requisite software in one place. This is one of those situations where you don't know what you don't know - so my ask is not going to be really clear.

I'd love for there to be a github page, so far I haven't found one. Have to say, I haven't looked very hard, maybe there is one!

That being said, I haven't quite figured out everything that I need. Once I do, I may create a page on github for this purpose.

At the moment I am diagnosing what appears to be a bad nvme memory. Not sure if it is the device, or the housing. The error is intermittent, so I am betting that it is mechanical and housing related.

 

[WobblyHand]

Nope. It's the nvme memory, swapped housings and kernel fails to find an operating nvme device. I found this using the command $ journalctl -k. journalctl is the newer, systemd version of syslog. All the disk readbacks for the device are zero. Also swapped cables - no difference. Then I put in a different nvme device and the nvme device was recognized. Major s%ckage, as I have two of these devices, now I don't trust the other one!

For the record, the bad nvme was a Western Digital 1TB WD Blue SN550 NVMe Internal SSD - Gen3 x4 PCIe 8Gb/s, M.2 2280, 3D NAND. I'll try doing a return to WD. They are allegedly warranted for 5 years. The time window for an Amazon return is long over.

May 30 11:29:25 rpi4 kernel: sd 2:0:0:0: [sdc] Synchronize Cache(10) failed: Result: hostbyte=0x07 driverbyte=0x00
May 30 11:30:28 rpi4 kernel: usb 2-1.4.4: new SuperSpeed Gen 1 USB device number 10 using xhci_hcd
May 30 11:30:28 rpi4 kernel: usb 2-1.4.4: New USB device found, idVendor=152d, idProduct=0583, bcdDevice= 2.09
May 30 11:30:28 rpi4 kernel: usb 2-1.4.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
May 30 11:30:28 rpi4 kernel: usb 2-1.4.4: Product: USB to PCIE Bridge
May 30 11:30:28 rpi4 kernel: usb 2-1.4.4: Manufacturer: JMicron
May 30 11:30:28 rpi4 kernel: usb 2-1.4.4: SerialNumber: 0123456789ABCDEF
May 30 11:30:28 rpi4 kernel: scsi host2: uas
May 30 11:30:28 rpi4 kernel: scsi 2:0:0:0: Direct-Access     JMicron  Generic          0209 PQ: 0 ANSI: 6
May 30 11:30:28 rpi4 kernel: sd 2:0:0:0: Attached scsi generic sg2 type 0
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] Unit Not Ready
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] Sense Key : 0x4 [current]
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] ASC=0x44 <<vendor>>ASCQ=0x81
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=0x00 driverbyte=0x08
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] Sense Key : 0x4 [current]
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] ASC=0x44 <<vendor>>ASCQ=0x81
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=0x00 driverbyte=0x08
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] Sense Key : 0x4 [current]
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] ASC=0x44 <<vendor>>ASCQ=0x81
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B)
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] 0-byte physical blocks
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] Test WP failed, assume Write Enabled
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] Asking for cache data failed
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] Assuming drive cache: write through
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] Optimal transfer size 33553920 bytes not a multiple of physical block size (0 bytes)
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] Unit Not Ready
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] Sense Key : 0x4 [current]
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] ASC=0x44 <<vendor>>ASCQ=0x81
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=0x00 driverbyte=0x08
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] Sense Key : 0x4 [current]
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] ASC=0x44 <<vendor>>ASCQ=0x81
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=0x00 driverbyte=0x08
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] Sense Key : 0x4 [current]
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] ASC=0x44 <<vendor>>ASCQ=0x81
May 30 11:30:36 rpi4 kernel: sd 2:0:0:0: [sdc] Attached SCSI disk