Samba, Windows, *nix, SELinux and you

Let’s face it: SMB/CIFS comes in handy plenty of times. Works great under Linux, works great with Windows, and if it’s configured properly it’s fun times for everyone. There are multiple potential behavioural problems though, mostly involving charsets.

Samba charset (and filesystem) setup

While I’ve been successfully using iso-8859-1 by default for a long time in my Samba servers, until recently I also used the same charset in the *nix filesystem, which brought all kind of oddities. 20/20 hindsight, as per usual. In smb.conf, you can configurn though you can specify different charsets to make everyone happy:

[global]
dos charset = iso-8859-1
unix charset = utf-8
preserve case = yes
short preserve case = yes
default case = lower
case sensitive = no

This has multiple advantages:

  • Windows gets its iso-8859-1 charset, and is happy;
  • Linux gets its utf-8 charset, and is happy;
  • preserve cases make sure that filenames aren’t changed while moving files across machines;
  • disabling case sensitive avoid weird behaviour under Windows.

This is the configuration I’ve been running for years, sharing files across Windows/Linux/OS X machines since Windows XP, and has been working perfectly.

Changing filesystem charset

If you’re switching the unix charset though, you might need to convert existing files/directories to the new charset. Cue in convmv:

convmv -f <from_charset> -t <to_charset> -r --preserve-mtimes <dir>

In my case:

convmv -f iso-8859-1 -t utf-8 -r --preserve-mtimes .

With this simple command I was able to mass rename thousands of files in one go. Note that you also need the --notest parameter to actually apply the changes, and not just list the files in a dry run.

SELinux

While playing around Centos I stumbled into SELinux problems I wasn’t prepared for. Consider the following:

[Share]
comment = Personal share
path = /home/<username>/share/
guest ok = no
browseable = yes
writable = no
create mask = 0660
directory mask = 0775
write list = <username>

Supposing the path actually exists, that the username is added to the samba user database and that the password is correct, everything should work fine. Except that it mostly likely won’t, because we need to configure SELinux on top of everything else:

restorecon -R -v /home/<username>/share/
chcon -R -t samba_share_t /home/<username>/share/

That’s all there is to properly setup and have the content accessible. Contrary to what other people, claims you don’t have to increase permissions in the samba path tree. A 0700 on the /home/<user> works perfectly fine.

CentOS 7 & DHCPv6

Sometimes instead of the usual SLAAC you might need or want to use DHCPv6. The dhcp package and a very short manual configuration will take care of everything for you on the server, but what about the client? What if you configured the DHCPv6 client but it is not working?

I setup both the server and the firewall, then I proceeded to configure the client, but it would always fail. No v6 traffic would hit the server machine, even though I added the DHCPV6C=yes on the client machine. After hours spent without a solution, I found a video with the answer: the problem is Network Manager! On top of adding the DHCPV6C parameter you also need to add NM_CONTROLLED=no. I ended up with the following /etc/sysconfig/network-scripts/ifcfg-ethX:

TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=no
IPV4_FAILURE_FATAL=no
IPV4INIT=no
IPV6INIT=yes
NM_CONTROLLED=no
DHCPV6C=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=no
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ethX
DEVICE=ethX
ONBOOT=yes
IPV6_PRIVACY=yes
ZONE=public

Perfectly disabled IPv4, IPv6 configured automatically, all fine and dandy. Happy fun times!

OSSEC troubleshooting

Today we continue the saga of things I was supposed to write down but didn’t, for reasons unknown. Suppose you migrated your OSSEC management server, or freshly installed what will be the new manager on a new OS. You import the keys, as described in my previous post, but the connection fails for one or both of these reasons:

  • ossec-remoted(1403): ERROR: Incorrectly formatted message from '<client_ip>'. – Pick your own adventure-style error message.
  • ossec-agentd(1407): ERROR: Duplicated counter for '<server_name>'. – Incorrect serials.

This has happened several times over the course of the last decade, due to client/server version mismatch, drive failures, and what have you. There’s a pretty brute-force way to solve these problems, though:

  • stop both server and client;
  • on the client, delete everything inside /var/ossec/queue/rids;
  • reimport the key on the client (unsure if this step is really needed);
  • start the server;
  • test that the client is working, via ossec-agentd -d -f.
  • if the client is working, start the service.

That’s it. There’s nothing that a good ol’ rm -rf * can’t solve.

Nvidia drivers and MSI support in Windows

Today I started searching for an old article of mine in regard to guest Windows VMs and the troubles with pass-through Nvidia cards. Picture me surprised when I found out that I never actually posted it, although the article has been in the back of my mind for the past two years or so. So, I’ll write it right now, since it contains valuable information that might help some people.

PCI pass-through

There are only a handful of problems with PCI pass-through of video devices:

  1. manufacturers are dicks. You can’t pass-through the first graphic card on consumer devices, because reasons. If you buy a workstation grade with the same hardware though, we’ll allow it.
  2. Nvidia is a dick. If the drivers on the guest sniff out that you’re running within a hypervisor, they won’t work. At all. They refuse to load.
  3. Nvidia is a dick. Although every card supports MSI mode as a replacement for line-based mode, every single time you install the drivers the MSI mode gets reset, as only the workstation/server grade drivers flag the system about message mode. You’re not using the card in a guest machine after all, right? Right?

So, here are fixes for the problems above, same numerical order:

  1. none. The best thing you can do is have GPU capabilities in the CPU. This could/should work (untested).
  2. there are ways to “unflag” a guest machine from the dom0. On KVM through QEMU you can specify a `kvm=off` for the CPU, or edit the machine with `virsh edit`.
  3. after the drivers are installed you can manually edit the Windows registry to enable MSI (also needs a reboot).

MSI and you

There are various arbitrary sources that can tell you why MSI is better than the default line-based counterpart, but when it comes to virtualization I can tell you the top reason why you want to switch to MSI: line-based is unstable. I’ve used my virtualized main workstation/gaming station for a while now, and the only times video card had troubles or the entire VM crashed, was because something between the drivers and the pass-through of the IRQ interrupts in line-based mode failed hard. Since the discovery of MSI I stopped having issues with the video card and everything runs butter smooth.

So, to recap:

  • Audio coming from the video card crackling? Switch to MSI.
  • Guest O/S crashing? Switch to MSI.
  • Video drivers throwing a fit? Switch to MSI.
  • Bored? Switch to MSI.
  • Switch to MSI.

Enable MSI

Checking is fairly simple, just open Computer Management’s Device Manager, and check if the NVIDIA Geforce <whatever> and the relative High Definition Audio Controller have a positive or negative value.

List by connection
Line-based
MSI based

If the value you see is greater than zero, you should switch to MSI. In order to do that, you need to open the device properties and find the device instance path:

With that in hand, you can open HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\PCI path in the registry, and follow the device instance path to find the following:

With MSI disabled you will notice that the MessageSignaledInterruptProperties key is missing, as you will need to create it along with the DWORD MSISupported set to 1.

That’s all there is to it. You can now reboot the system and the drivers will use MSI mode. Any audio crackling coming from the monitors will be gone, and everyone will rejoice.

Import disk image into XCP-ng/XenServer

Importing a disk image into XCP-ng/XenServer is rather easy, you just need to find the correct VDI and import it using the appropriate `xe vdi-import`. In my case I wanted to test the current OpenWRT release, so:

# xe vdi-list
uuid ( RO)                : <vdi-uuid>
          name-label ( RW): OpenWRT
    name-description ( RW):
             sr-uuid ( RO): <sr-uuid>
        virtual-size ( RO): <size>
            sharable ( RO): false
           read-only ( RO): false

# xe vdi-import uuid=<vdi-uuid> filename=openwrt-combined-ext4.img

That’s it: OpenWRT is installed on Xen and already bootable. At this point you can even mount the OpenWRT drive and resize its partition to fully use the remainder of the space you allocated.

Auto IP assignment to a dom0 virtual interface – Part #2

As we saw eariler, with careful configuration and the use of my small service, we’re able to remote in the dom0 no problem, but the main pitfall I didn’t realize was there until a couple days ago, is that in that iteration of the service, now updated, we lost console and performance monitor for most of the VMs spawning after the UTM. I tried asking for pointers, but given the “experimental” nature of my setup, I was left on my own.

The problem

First of all, while I was setting things up I left an ip outside of the class I was using to the management interface, in order for it not to interfere. This resulted in consoles not working for about any VM, since instead of a specified class it was using 192.168.4.0:

# xe console-list
uuid ( RO)             : <uuid>
         protocol ( RO): RFB
         location ( RO): https://192.168.4.x/console?ref=OpaqueRef:<ref>

Reconfiguration through `xsconsole` or Center don’t work, obviously, so the only thing left to do was to restart in safe mode and change to the proper subnet. So far so good.

Upon restart everything seems working, up to the time I actually start a different VM:

# xe console-list
uuid ( RO)             : <uuid>
         protocol ( RO): RFB
         location ( RO): https:///console?ref=OpaqueRef:<ref>

This time everything was configured properly, but it still wouldn’t work. As it turns out, somewhere between starting the VMs and setting up the network, the bridge loses its ip. Xapi doesn’t care in the slightest what IP you’re connecting to, so when it has to pass a link to a management section, it just looks up its management network bridge for the address binded to it and generates it.

The solution

The solution is pretty simple: on top of adding a /24 mask to the VIF you’re connecting to, you also need to add the same ip with a /32 mask to the management bridge:

# ifconfig
xapi1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.1  netmask 255.255.255.0  broadcast 0.0.0.0

xenbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.1  netmask 255.255.255.255  broadcast 0.0.0.0

Thanks to [SkiD] for moral support and reminding me that this configuration is actually valid, since in routing terms `xenbr0` doesn’t go anywhere.

Auto IP assignment to a dom0 virtual interface

Let me take you on a journey through the Hypervisor valleys, across the domU PCI NIC pass-through river, that resolves into the land of service coding and dynamic dom0 ip binding.

The long, anti-TL;DR preamble…

Long time readers of this blog, and I mean real long time since my last post is a few years old, know that I’ve worked with hypervisors for quite a bit. What you don’t know is that, contrary to what I mentioned in a previous post, I actually spent the last couple years using a virtual machine as my main desktop environment, and I loved it. But I’m a gamer, and VFIO drivers have problems including heavily suffering from bufferbloat, so I’m planning to go back to bare metal, and that’s the main reason that made me bring in a new server to the house.

I’ve spent the last few years working with different kinds of hypervisors, KVM being the latest for the past year and a half, but all in all no matter the KVM, ESXi and Hyper-V of the world, despite what Citrix did to the poor thing, I missed XenServer the most. So, imagine my joy in realizing XCP-ng existed and that it was good. Marvelous.

So, why did I want a new server to begin with?

Firewall/UTM/Border gateway. I’ve finally been allowed into the realm of gigabit fiber network (been trying and praying and contacting people for the past 17 years), but the modem/router by my ISP obviously sucks. For the same reason ISPs provide upload speed that are 20% your download speed (you must not host your own servers/clouds) they also give you a class /64 IPv6 subnet (YAY!) but the router can’t handle firewalling IPv6, barely straight IPv4 NAT’ting, so if you use IPv6 you’re completely exposed to the interwebs. “Wait, what the actual f***?” I hear you say, and to no one’s surprise, the same thing I asked myself.

I love my 64 GB RAM daily workstation, with all of its 12 cores good-y-ness, but KVM can’t manage it. I’ve been using my main workstation as a Linux HTPC, Windows workstation, and as a multi Linux VM server box. It works great, mostly, but I no longer have any use for the HTPC part of the equation, I have issues with bufferbloat caused by the VFIO drivers, and the PCIe passthrough eats about 10% of my graphic card performance. I have a 1080ti, so that’s no biggie, but all things considered I’d be better offloading the few VMs I run on my workstation elsewhere. I might still try different things with KVM or XCP-ng when my virtual servers are safe on the other machine, but that will go into a new post, eventually.

Big change in the office topology. While for the past 25 or so years everything has always been mostly in a short Cat5 range, now the machines are running on multiple floors. A main server able to serve the needs of everyone in-house and as close to the internet access as possible, is a win-win all around.

These are the main three reasons that made me spend an ungodly amount of money, considering what this server will be required to do. Initially I was hell bent on buying an Intel NUC or some mini box off aliexpress, and I almost did it, but I ended up designing a box with the following features:

  • CPU with an Intel GPU. No ROM locking for graphical cards in the first PCIe slot, should I need to upgrade the box for video transcoding or 3D rendering.
  • 32 GB of RAM. I had 4x8GB Kingston HyperX Beast laying around doing nothing.
  • 5 Intel NICs. 1 onboard, plus a PCIe network card with 4 Intel NICs.
  • Full VT-x and VT-d compatibility.

Let’s just say that while I started embracing Jeff Atwood’s idea of the scooter computer, I ended up with a heavy quad which costs three times as much, as per my usual. Again, to no one’s surprise.

So, armed with my quad computer — which incidentally is also quad-core, though hyperthreaded — I installed XCP-ng fairly effortlessly, only to find out that for once in my life everything works, including the second-hand parts I bought on eBay. My, oh my, there’s a first for everything. Well, everything except for the damned Kingston Beast, which won’t work at their labeled clock speed if their life depended on it. But a more relaxed XMP profile fixed that well known issue — G.Skill for LIFE! —.

The server’s network architecture

In a typical day of a typical server with a typical configuration, the architecture would look something like this:

A dedicated physical interface isolated from the rest of the LAN that, at times, is also bridged to the other networks for the traffic portion not inherent to the dom0, while the dom0 manages through virtual bridges the traffic coming from the LAN and from the virtual machines, while at the same time routing everything to the WAN. In this case the dom0 acts as a router, while an eventual domU UTM acts as a service firewall.

Cue in my sweet madness:

Since I wanted to use my NICs to the best of my abilities along with traffic shaping, virtual interfaces aren’t good enough. So, armed with patience and a sprinkle of carelessness and reckless abandon I proceeded through trial and error — mostly locking me outside of the dom0 — to passthrough every single physical NIC to the UTM. I kept one physical interface connected to the dom0 while I was setting up the VM to receive the network cards, and it almost went flawlessly until I inverted the order of operation and rebooted the machine without accepting the changes on the UTM. Oops! Anyway, this setup has several advantages:

  • Full control and speed of the Intel NICs straight on the UTM.
  • Traffic shaping.
  • Tightening up the dom0, which doesn’t have physical access to the network anymore.
  • One less layer of communication between the WAN and the domU appliances.

All of these advantages for mostly no disadvantage:

  • If the UTM VM goes down, your network is down and so is your access to the dom0.
  • Some of the current tools become unusable.
  • There is no one to set up the dom0 virtual interface network once everything started.

What’s there not to love?

The passthrough journey

I had my objectives set, and so I started scouring the interwebs for answers. To my surprise the entire proper set up is summarized with six commands tops:

  • lspci | grep Eth. Find the NIC targets.
  • xe vm-list. Find the target VM UUID.
  • /opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(01:00.0)(01:00.1)(01:00.2)". Enable passthrough on the dom0.
  • xe vm-param-set other-config:pci=0/0000:01:00.0,0/0000:01:00.1,0/0000:01:00.2 uuid=<vm_uuid>. Passthrough the PCI devices to the target VM.

And that’s it with 4 commands. If you’re running something like pfSense, as you can read on this GitHub wiki page, you need to work a little harder since it doesn’t handle well (read, at all) empty checksums on Ethernet packets, so you also need the extra 2 commands:

  • xe vif-list vm-uuid=<vm_uuid>. Find the virtual interfaces UUID connected to the target VM.
  • xe vif-param-set uuid=<vif_uuid> other-config:ethtool-tx="off". Disable the transfer offload for virtual interfaces.

On pfSense, you will want to increase the performances by paravirtualizing some devices, thus (as you can read here in this Netgate forum post) while on the pfSense VM issue these commands:

  • pkg install xe-guest-utilities. Install Xen-aware drivers.
  • echo 'xenguest_enable="YES"' >> /etc/rc.conf.local. Enable Xen guest agent.
  • ln -s /usr/local/etc/rc.d/xenguest /usr/local/etc/rc.d/xenguest.sh. Create the link necessary for the service to start at boot.

That’s it, you’re done, you can reboot straight into the VM with hardware NICs passed through. Except that if you do, you’ll lock yourself out of your own server. Fun times!

Coding our way through

The thing that took me the longest to figure out was that for example tools like XCP-ng Center put the host in maintenance mode and then issues a reboot, and after it booted back restores the state in which the host was before maintenance. Except that, if you followed me so far, there’s no going past maintenance mode, because as soon as the UTM goes down so does the link with the dom0, so the server is stuck with all its VMs shut down and sits there wondering what went wrong with its life, questioning its life decisions.

That’s a minor issue, we can still SSH into the dom0 and use xsconsole to do the same thing without locking us out, but it took me a few tries and some reverse engineering to figure out that no, some of the XCP-ng Center features are 100% unusable in this scenario. You know what? It’s ok, I can live with that.

What I can’t live with is a dom0 perpetually out of reach, so I headed out to Server Fault to find people who actually knew better than me. Surely I’m not the first to encounter this problem, right? The silence was deafening.

“Well,” I told myself, “explorer of the unknown is my middle name” — and what a strange middle name to give your first born — so I started hacking something up. I was tentatively crafting something in bash script, but I soon realized that it was more trouble than it was worth. I also found out that I had two tools at my disposal on the dom0:

  • Python 2.7.
  • XenAPI python module.

This is when I struck gold. Through time-vm-boots.py and monitor-unwanted-domains.py I figured out that I would have been perfectly able to start a service, let it sit there, and dynamically attach an IP to the virtual interface that spawned with the VM. Easier said than done, but possible.

So, armed with the XenAPI documentation for virtual machines, virtual interfaces, networks, and later physical block devices, I crafted my own service.

The only thing left to do was let it start at boot, but no amounts of crontab seemed to work, so I decided to make it into a fully-fledged service with the help of systemd. Except, of course, for the fact that the last time I set up a service like this was many, many moons ago. But this is why we have the interwebs, innit? So I took it to the HTML’d systemd man and to RedHat systemd training material, and my eyes feasted on the latter, because it was actually formatted for humans.

A few tests, commits, and times locking myself out later, I finally had a service worth using at my disposal. It isn’t perfect, there’s (at the time of writing) no native python way coded in to ping v6 targets, and there’s no way to unlock a currently running dom0 which went in maintenance mode, but it grew way more than anticipated, and works fantastically well.

17:15:36 xcp vipb.py: Initialised to add v4 192.168.0.2/24 via 192.168.0.1 to
                      pfSense's network dom0 using /usr/sbin/ip (XAPI timeout: 30.0s)
17:15:36 xcp vipb.py: Adding 192.168.0.2/24 to xapi1
17:15:36 xcp vipb.py: Adding default route via 192.168.0.1
17:16:19 xcp vipb.py: Plugging SR "ISO Repository"'s cifs "//192.168.0.3/ISO" PBD

I’ll link you back to the vif-ip-binder project on GitHub, with full sources and a bit of technical details, in case you previously overlooked it in the article. And with this, just like Chamber sang, our journey finally ends, a tale of true love.

Migraine free Java SSL management

I recently had the need to turn a JAVA web application into SSL only. While the configuration is almost painless (-Dhttps.port=443 -Dhttp.port=disabled), the certificate management wasn’t quite as effective, due to a certain lack of clear documentation (where have I heard that again?).

We start assuming you already have your OpenSSL generated key/crt files, because that’s what happened here, but stay tuned:

With this done you have a PKCS12 KeyStore you can use for the web server, much like the PEM/KEY you would use in Apache. Then, you use this configuration to properly load it:

It may seem like nothing, but the keyStoreType part made the difference between a proper certificate chain and an unsigned chain localhost/localhost, which caused me major migraine for the good part of an afternoon.

With this said and done you can start the application/server and see to it that the certificate now lists itself properly and with all the needed CAs attached to it.

Self-Signed Certificate with Subject Alternative Names (SAN) [AntiFUD]

Wrangling obscure OpenSSL functions to create and publish SSL certificates has always been kind of a mess. If you want(ed) to create a valid self-signed certificate for multi domains or, at least, example.com and www.example.com, you most likely were out of luck.

There is a lot of wrong or partial documentation on the subject, but is… well… wrong and/or incomplete. It is thus time for another episode of AntiFUD.

The problem

You have multiple paths of the same website to cover for, but a single CN. If you use example.com then www.example.com will result in invalid SSL certificate, and vice versa. Suppose you have the following domain names:

  • example.com
  • www.example.com
  • *.user.example.com

In such a scenario there is no real victory no matter what you choose to use as a CN: the most used wildcard CN, *.example.com, is of no use either because it matches with www.example.com and user.example.com, but not with username1.user.example.com. The only way to address all these issues is to create and sign a X.509 v3 SSL certificate, to allow SAN. The SAN extension has been introduce to resolve all of these problems, allowing the validity of multiple domains/subdomains within the same certificate.

Creating the certificate

We have to start by creating an alternative configuration file to use with OpenSSL, and list the server names we need. As mentioned below we also have to enable the usage of v3 extensions.

We can now edit the file and adjust as needed:

In the default file, parameters such as req_extensions and keyUsage are commented out, while subjectAltName is missing. We have to add it to v3_req and v3_ca, and create the respective section. It can be created anywhere in the file, but it is generally appended to the bottom. Since the CN is (or, at least, should be) ignored in the presence of SAN, we insert all the names in the alt_names field.

With the configuration in place we can now create the certificate:

The deviation from the standard procedure is the addition of the v3 during the CA sign. We do this by using -extfile example-com.cnf to use the custom configurations, and specifying -extensions v3_ca to make sure SAN are passed through and saved in the signed certificate.

To make sure it worked you can do the following:

The only thing left to do is to set up the certificates in the server, and everything will work as intended.

RSA/DSA ssh(d) keys, a synthetic guide

There is a lot of useless and cryptic information in regard to any type of encryption, typical as per USA’s FUD standards. I’ll post here a synthesis of the steps necessary to wave plain text password logins goodbye.

I’ll assume you already have the private/public key couple by now, if not you can use puttygen. This topic is well covered, although they have a tendency to suggest a low level of encryption. Isn’t it strange how for apparently “anybody” an 8 letter password, or a 2048 bit key, is enough for everyone? For the record, I used a 4096 bit DSA key.

I will also assume that you’re setting up a server on Linux, so ymmv.

Coming back to the topic at hand, you have a private key, that you use to login from your computer, and a public key that you will deploy to one or more PCs/servers. The public key will probably look like this:

This won’t work in most cases, as SSHD expects a certain format. You will then have to convert that key into this:

The beginning of the line is ssh-dss for DSA keys, ssh-rsa for RSA keys. With this line of text in hand, you can open ~/.ssh/authenticated_keys on your servers, copy the key data into it, and save.

The last thing to do is to reconfigure the sshd.

While checking the configuration I noticed that the ephemeral key size (ServerKeyBits) was defaulted to 1 kilobit. ONE FREAKING KILOBIT. To give you a comparison, in 2002 on IRC channels we used DH with 2048 bits of encryption. That’s 13 years ago. For chat. You might want to turn it up several notches.

For the server to actually use the key you provided, you will need to uncomment the AuthorizedKeysFile, keep in mind that the path may differ. It could be .ssh/authorized_keys on CentOS, %h/.ssh/authorized_keys on Ubuntu, so on and so forth.

AFTER you made sure you can actually log in with your DSA/RSA key, you will disable plain text authentication by uncommenting AuthorizedKeysFile and setting it to no.

This is all the black magic involved in it, without the convoluted mess that always surrounds OpenSSL/SSHD documentations.