by Pedro M. S. Oliveira | Jul 10, 2017 | Linux
Why care about IO performance?
Recently I’ve acquired some new hardware and I want it to perform as fast as possible. The setup is quite trivial for a home desktop nevertheless I wanted it to excel on IO performance as it will be use as my backup server too. A common way to improve performance is by adding a cache system, this applies to many things in IT and block devices are no exception.
The relevant hardware components for this post are the 7x 2TB drives and 1x NVME card. The setup is not ideal as the models are not all the same, some to perform better than others some are newer and others older. Nevertheless money and storage capacity was important also I wanted to use them anyway. Security is also very important, so all the data written to these drives (including the NVME card) must be encrypted. On the other side I want to be able to expand the raid devices when the time comes so I also use LVM, as file system I use XFS with the default settings.
You may wonder why didn’t I use a simpler setup with BTRFS or ZFS? Mostly because I wanted to use raid 5 or 6 and on BTRFS the stability is still and issue on this form of raid. On the other hand with ZFS it would be difficult to grow to the pool in the future.
The logical setup is as follows
- 1x raid 5 with 6 drives (+1 hot spare)
- LVM on top of the raid device
- Cache device or Logical volume
- Block encryption layer – LUKS
- File system – XFS
The hardware list
- NVME Samsung SSD 960 PRO 512GB
- ST2000VN004-2E4164
- ST2000VN004-2E4164
- ST2000VN004-2E4164
- WDC WD2003FYPS-27Y2B0
- WDC WD200MFYYZ-01D45B1
- WDC WD20EZRX-00D8PB0
- WDC WD2000FYYZ-01UL1B1
The NVME device is used both to the OS, Home, etc, but it does contain a LVM logical volume to be used as cache for the raid device. The number of IOPS / bandwidth the NVME is rather high, it goes all the way up to 440.000 IOPS and a bandwidth of 3.5GB/s, which is quite insane and I won’t be able to exhaust with my day to day use, so it can spare a few IOPS to make my backups go a bit faster.
I’ve tested bcache and lvmcache, as a benchmark tool I’ve used iozone. I’ve done the tests with 256kB,1MB,8MB block sizes, the test file is 96GB (as it needs to be bigger than the total ram amount 64GB).
The initial test was made using the full setup without any caching system, it will be used to set a base of comparison.
Each test was done with 3 different block sizes 256K, 1MB, 8MB, test settings and cache mode for all the test is “writeback”:
- md device, raid 5
- lvm volume
- luks
- xfs
Results
Using no cache
Test setup
- MD Raid 5
- LVM lv data
- Luks
- XFS
Using lvmcache
Test setup
- MD Raid 5
- LVM lv data
- LVM lv meta
- LVM lv cache
- LVM lv cache pool
- Luks
- XFS
Using bcache
Test setup
- MD RAID5
- LVM LV data
- LVM LV cache
- bcache volume
- Luks
- XFS
Test results – Benchmark graph
Conclusions
In what regards to overall performance the outcome is not as expected. LVM cache really didn’t seem to improve the system performance. In some of the tests it was quite slower than the no cache mdraid and in some other just slightly faster. Nevertheless bcache did show real improvement being faster in all the tests, some by more than 30%.
Although bcache improves the system, it’s also the most difficult system to setup, lvmcache is totally integrated in LVM tools and in the kernel, bcache requires the installation of bcache-tools as it’s not a default on most distributions.
If you fell comfortable with Linux, block devices, mdraid and LVM I would recommend it without worries, if you’re not familiar with this set of tools I would recommend you to test your setup before you run it in a server / desktop environment.
The performance benefits are worth the extra work.
Test raw report files
Bellow are the iozone generated reports and the ods spreadsheet I used to build the graphs.
iozone_test_without_cache
iozone_test_with_cache_bcache
iozone_test_with_cache_lvmcache
Benchmark results
by Pedro M. S. Oliveira | Jun 25, 2017 | Linux
If you don’t have LLPDd available you may use tcpdump to get all LLDP information. Bellow are a few examples on how you can do it.CDP
## This will often show you the Cisco chassis switch, then use your firms asset management software to find the upstream switch.
## -s 1500 capture 1500 bytes of the packet (typical MTU size)
## ether[20:2] == 0x2000 – Capture only packets that are starting at byte 20, and have a 2 byte value of hex 2000
interface=eth0 ; tcpdump -i ${interface}-v -s 1500 -c 1 'ether[20:2] == 0x2000'
LLDP
## Switch:
interface=eth0 ; tcpdump -i ${interface} -s 1500 -XX -c 1 'ether proto 0x88cc'
## Port and CDP Neighbor Info:
interface=eth0 ; tcpdump -i ${interface} -v -s 1500 -c 1 '(ether[12:2]=0x88cc or ether[20:2]'
If you need more info about CDP or LLDP have a look on the links bellow.
CDP stands for
Cisco Discovery Protocol, which is a layer 2 protocol and is used to share information about other directly connected Cisco equipment (WikiPedia). LLDP stands for
Link Layer Discovery Protocol and replaces CDP. LLDP is a vendor-neutral Data Link Layer protocol used by network devices for advertising of their identity, capabilities and neighbours (WikiPedia).This is useful to find out what
VLAN your network interface is connected to (assuming that your using tagged VLANS), or what port am I plugged into on which switch.
by Pedro M. S. Oliveira | Jun 19, 2016 | Daily life, Linux
I use digiKam as my primary photo manager, it’s a great professional tool for those work with media. Maybe you are one of these and you didn’t even noticed, look at your mobile phone probably you have hundreds (maybe even thousands) of pictures on it. If you sum the ones you take with your camera it will be even more, so after a few years some of us may have more than 100.000 pictures.
This is why you need a photo manager and I would strongly recommend digiKam. I’ve been using it for years and I love it.
One part of taking pictures is be able to share them, digiKam supports exports to several social media web sites; Facebook, Igmur, Twitter, Flickr, etc.
Nevertheless the integration with Facebook is not working properly, and I doubt that the digiKam developers can do something here as it looks more of a Facebook issue. I’ve came out with a workaround that may help go around this issue if want to use the Facebook export function. Just follow these easy steps (I’ve done this with digiKam v5, but it will also work on v4):
- With your default browser log in to Facebook.
- Open digiKam from a console, just type digiKam on the command line.
- Inside digiKam go to menu export, choose export to Facebook.
- On the export dialog press change account.
- You’ll see that in the browser it just states success, nevertheless you don’t see the authentication link that you need to use to paste into the plugin dialogue.
- On the console where you started digiKam you have some of the debugging info of what is happening
- digiKam uses OAuth to authenticate with Facebook so look for a line like:
- OAuth URL: QUrl(“https://www.facebook.com/dialog/oauth?xxxxxxxxxxxxx
- Copy the content of that line starting on “https://…token” (without the quotes)
- Enable debug mode on your browser (I use Firefox CTRL+SHIFT+C or F12)
- On the debug window select network and persistent, see selected options on image bellow:
- Paste the URL that you got on this list step 7 on the browser
- On the debug window you should be able to see an entry like the one bellow, with a right click choose copy location
- Go to the digiKam Facebook export plugin window and just paste.
- YOU’RE DONE
Now you can share all those pictures to world 🙂
Just select the pictures you want, use the export plugin, choose your Facebook album (or create a new one within digiKam), choose to resize (or not the images). If you create a new album be careful with the the permissions you wish for the album, on the Add Album option you’ll be able to select who do you wish to share your pictures. By default all your friends will be able too see it, but this may not be right for all the albums so just choose careful your options, usually I choose only me and then on Facebook page I add who I want to share it with.
More about digiKam
- A professional digital photo management application for KDE (it will work on Linux, Windows and MacOS), which makes importing and organizing digital photos a “snap”. The photos are organized in albums which can be sorted chronologically, by folder layout or by custom collections
- Powerful, fast and automated tagging
- An easy-to-use interface is provided that enables you to connect your camera directly and preview, download and/or delete your images
- A Light Table function is provided. This tool assists artists and photographers with reviewing their work ensuring the highest quality only
- Common filters are available out of the box, with side by side previews
- Extensible by the use of plugins and recipes
Hope you enjoy this tool as much as I do.
Have fun,
Pedro Oliveira
by Pedro M. S. Oliveira | Feb 29, 2016 | Linux
Sometimes a software/hardware error will lock your SATA device/controller. You may just reboot your OS and wish for the best, but you may also only reset the affected device, not the full system, bellow is how you can achieve this.
Before you apply the following commands make sure you are not using the SATA/SCSI devices (ex. mounted filesystem, part of a RAID, LVM, BTRFS or ZFS volume). If the device is still in use please take action to stop it before you delete the device node or reset the SATA bus.
A device name of /dev/sdb is assumed.
- Find out which controller the device is attached to (we’ll need this later):
# readlink /sys/block/sdb/device
../../../1:0:0:0
The interesting part if the answer is host1, which identifies the controller.
# echo 1 > /sys/block/sdb/device/delete
This will remove the device from the bus (logically). Look in dmesg for confirmation.
# echo "- - -" > /sys/class/scsi_host/host1/scan
host1 is the identifier from step one. Again, dmesg should show the device being rediscovered.
Have fun,
Pedro Oliveira
by Pedro M. S. Oliveira | Mar 13, 2015 | Linux
Several time I’ve been asked how to rename a network card label on a Linux system, this can be achieve by following the steps mentioned bellow.
Stop networking
service networking stop
The network kernel module needs to be removed, but first you need to identify the right kernel module
sudo lspci -k | grep -i -A4
ethernet 04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 07) Subsystem: Sony Corporation Device 9099
Kernel driver in use: r8169
Kernel modules: r8169
Reload rules
udevadm control –reload-rules
Edit udev rules by editing the /etc/udev/rules.d/*persistent-net.rules, in the example bellow we will set previous eth0 to eth5
# PCI device 0x10ec:0x8168 (r8169)
SUBSYSTEM==”net”, ACTION==”add”, DRIVERS==”?*”, ATTR{address}==”d4:3d:7e:xx:xx:xx”, ATTR{dev_id}==”0x0″, ATTR{type}==”1″, KERNEL==”eth*”, NAME=”eth5″
Remove the module in question
rmmod r8169
I’m using an realtek chip on my Vaio laptop, you should replace r8169 with the module in your system
Request device events from the kernel
udevadm trigger
Start network
service network restart
Et voila! You should have now a network card named eth5 on your system.
If you want to learn more about udev and udevadm check the info here or just type man udevadm on the command line.
Have fun,
Pedro Oliveira
