Archive for the ‘Rants and Essays’ Category

Automatically ending a process in Linux

I have a Python script that I want to run from one console, and sometimes I want to be able to kill it from another console login (via SSH or direct to the console). To do this easily I simply wrote a bash script that uses grep and awk and then passes the results to a ‘sudo kill’ command like this:

sudo kill $(ps aux | grep 'sudo python script1.py' | awk -v i=1 -v j=2 'FNR == i {print $j}')
sudo kill $(ps aux | grep 'python script2.py' | awk -v i=1 -v j=2 'FNR == i {print $j}')
cp /var/log/security_camera.log ./

However, if you have a particularly stubborn script, you may need to use the KILL signal. To do that just specify the KILL signal like this:

sudo kill -s KILL $(ps aux | grep 'sudo python script1.py' | awk -v i=1 -v j=2 'FNR == i {print $j}')
sudo kill -s KILL $(ps aux | grep 'python script2.py' | awk -v i=1 -v j=2 'FNR == i {print $j}')
cp /var/log/security_camera.log ./

Only use the latter if you absolutely have to. There are actually a lot of signals in between the default SIGTERM signal. See:

man kill

to see the system documentation on which signals you should use when. Or just Google it. 🙂

As with most things in Linux, this is only one way to do things. But it works great for me.

This post was inspired by this Linux comic I saw on Google+ ✌  The code is all written by Leland Green (me). I take absolutely no responsibility with what you’re about to do with it! 

An Introduction to SSL, TLS and HTTPS

shieldSecure Socket Layers (SSL) refers to a set of cryptographic protocols originally developed by Netscape for the purpose of securing communication between endpoints in a network. Due to security vulnerabilities, all versions of SSL have been deprecated and use of Transport Layer Security (TLS) is strongly advised. Because TLS is essentially a newer version of SSL, the term SSL is commonly used to mean either SSL or TLS.

Secure communication with a website is accomplished by means of the HTTPS protocol which is simply the use of SSL/TLS to encrypt HTTP messages. All modern browsers are capable of HTTPS communication, but it must be manually enabled on the website before it can be used.

To enable HTTPS for a website, an X.509 certificate is required. These certificates are typically purchased from a Certificate Authority (CA) such as Symantec, VeriSign, Thawte or GoDaddy and can be fairly expensive. An X.509 certificate contains information about who it was issued to (usually a website domain name), who it was issued by (usually a CA) and a public key which can be used for encryption and decryption. The public key in the certificate is mathematically related to a private key known only to the owner of the certificate. Information encrypted with the public key can only be decrypted with the private key and vice versa. This is known as asymmetric key encryption.

When a website resource is requested using HTTPS a SSL/TLS handshake must occur before any information can be exchanged. The purpose of this handshake is to verify the identify of the website, establish which cryptographic algorithms to use (the cryptosuite) and agree upon a shared master key both parties can use for encryption and decryption. In general, the process consists of the following steps. For a more detailed explanation, Chapter 4. Transport Layer Security of High Performance Browser Networking by Ilya Grigorik provides an excellent description.

  1. A TCP/IP connection is established.
  2. The browser sends information about which protocol versions and ciphersuites it supports.
  3. The server selects a protocol version and ciphersuite and attaches the website’s X.509 certificate.
  4. The browser validates the certificate, generates a master key and sends it securely to the server by encrypting it with the public key in the provided certificate.
  5. The server decrypts the master key with it’s own private key and notifies the client that it is ready to proceed.

It is worth noting that the certificate’s public key is only used once (to encrypt the shared master key in step 4 above). Although it would be possible to use the certificate’s public key to encrypt and decrypt all data sent to and from the server (eliminating the need for a shared master key altogether), it is not practical. Asymmetric key encryption is significantly slower than symmetric key encryption. Therefore, in order to maximize performance, asymmetric key encryption is used only in the handshake and symmetric key encryption is used for the remainder of the connection.

Since encryption alone can only guarantee privacy, another important aspect of the handshake is the certificate validation process. This step verifies the identity of the website and ensures that the browser is not communicating with an impostor. Certificate validation is based on a system of trust. Every X.509 certificate is signed by another X.509 certificate. This signifies that the owner of the signing certificate trusts the owner of the signed certificate. In this way, any given X.509 certificate forms a node in a chain of trust. The root certificate in every chain of trust is self-signed and must be trusted explicitly.

Typically, website certificates are leaf nodes in the trust chain and CA certificates are root nodes. Most browsers ship with a list of trusted root certificates from well known and trustworthy CA’s. Most operating systems also ship with a similar list of trusted root certificates and also provide a way for users to add new certificates to this list. In general, certificate validation in the SSL/TLS handshake simply verifies that the certificate presented by the website matches the domain name that was requested, that is has not expired or been revoked and that it chains up to an explicitly trusted root.

If any part of this process fails, the browser will inform the user that there is a problem with the certificate and may also provide an option to continue. When this occurs, there is no guarantee that the connection is being made to the intended website or that any information exchanged will be private. Although this sounds very serious, it may often be acceptable to proceed despite the warning. Ignoring a certificate validation warning is no less secure than accessing a website with the HTTP protocol (no security at all). Although it is not ideal to access any site over HTTP, it is nevertheless common practice and often the only option available. For websites that require the exchange of financial, personal or otherwise private information, a valid HTTPS connection should always be used.

In the end, the most important thing to understand about these protocols is what aspects they guarantee about communication.

  1. Confidentiality – Communication is private. This is achieved by encrypting all data with a key known only to the communicating parties.
  2. Integrity – Communication cannot be altered without detection. Although not discussed above, a Message Authenticate Code (MAC) is included in every exchange. This allows the receiver to verify that the message was not modified or corrupted in any way since the MAC was calculated.
  3. Authenticity – Communication is occurring with the intended party and not an impostor. This is verified during the certificate validation process of the SSL/TLS handshake. A fully trusted certificate implies that the owner is who they claim to be and that they (and no one else) control the certificate’s private key.

The Ghost of Roboduck Jones

An open letter to BL from T.

So I was sitting here reminiscing and guess what popped into my head? Roboduck! Do you guys remember that project? Just a little robotic duck that swam around teasing other ducks to suck on a shotgun. I think it sank. There may have been plans for a flamethrower. There should always be plans for a flame thrower. I wish I still had the pics. Alas but no, nor does wayback machine but I was surprised to discover that a new group holds bltlabs.com. How interesting.

<30 minutes pass>

Well look who I found hanging out in an archive of jointsandjams.com. I feel better, but now I want to work on jointsandjams.com again.

roboduck

In case you are interested, I have cobbled together a little Facebook community page for section9 and shared a few of our projects of the years. How the hell did we ever have time for any of this? The newfangled Raspberry Pi’s are just not the same. It’s not fun unless it’s a hand grenade. You can quote me on that.

https://www.facebook.com/section9.space

Moving Your WordPress Website to a New Web Server

This site is for those who are curious about how things work so I’m going to begin with some background. If you’re just interested in getting the job done, skip down to the How-To section.

Background

A website is a set of files that exist on a computer that is configured to run a special kind of software called web server software.  The two most popular web server software packages are Apache Software Foundation’s Apache HTTP Server and Microsoft’s Internet Information Services (IIS). Computers that run web server software are often referred to as web servers even though they usually run many other types of software as well. A web server works by listening for HTTP requests from web clients. There are many types of web clients, but the most common kind is the web browser (Chrome, Firefox, Internet Explorer, etc.) When you type in an address or click a link, your web browser sends an HTTP request for a specific file on a specific web server. When a web server receives such a request, it will deliver the content of the requested file to the requesting web browser. When the requesting web browser receives the content of the requested file, it will interpret it as best it can and display it to the user. There are many types of content and many ways in which such content can be parsed and interpreted, but this is the basic process in a nutshell.

All web servers have an Internet Protocal address (or IP address) which is like a mailing address for computers. IP addresses, however, are hard for humans to work with. Instead, we use domain names, which are simply aliases for IP addresses. For example, if you run the command ping www.google.com, you should see output similar to the following.

Pinging www.google.com [74.125.225.241] with 32 bytes of data:
Reply from 74.125.225.241: bytes=32 time=30ms TTL=55
Reply from 74.125.225.241: bytes=32 time=32ms TTL=53
Reply from 74.125.225.241: bytes=32 time=30ms TTL=55
Reply from 74.125.225.241: bytes=32 time=31ms TTL=53

The ping command tells us that we are able to communicate with the computer at IP address 74.125.225.241 using the domain name (or alias) www.google.com. This may seem complicated, but it is very useful when you want to do something like move a website from one web server to another. In our case, we wanted to move Section9 from a web server owned by A2Hosting to a web server owned by Arvixe Web Hosting. We were able to do this with almost no downtime simply by installing a copy of Section9 on the new Arvixe web server and then when it was ready, we simply went to our domain name registrar (GoDaddy) and updated the configuration of our domain name (section9.choamco.com) to point to the IP address of our new web server.

How-To

So from start to finish, a WordPress website can be moved to a new web server by following these simple steps.

  1. Backup up the old WordPress database. This is where your posts, pages, comments, menus and settings are stored. A good tool to use for this is the Export operation in phpMyAdmin. Be sure to export using UTF-8 character encoding.
  2. Backup the old wp-content folder. You can find this folder in your WordPress root location. This is where your plugins, images and other media are stored.
  3. Install WordPress on the new web server.
  4. Restore the old WordPress database on your new server. A good tool for this is the phpMyAdmin import operation. Be sure to import using UTF-8 character encoding.
  5. Restore the old wp-content folder into the new WordPress root location.
  6. Using the administration tool of your domain name registrar service, update the name servers for your domain name. Each domain name requires at least two name servers. These values are used by web clients to find the actual IP address of the server associated with your domain name. When you created an account with your new web hosting service, you should have received two name server names and their IP addresses. They often look like ns1.yourhost.com and ns2.yourhost.com. Replace the current name server values with the ones given by your new web hosting service. Once you make this change, there will be a 2+ hour delay, depending on your domain name registrar service and then all traffic to your domain name will be directed to your newly moved website.
  7. Test your website. If the links are broken, you may need to go to Settings->Permalinks and Save Changes in order to regenerate your permalinks. If there is missing data or your posts have been truncated, you probably have a character encoding mismatch. Re-export the original WordPress database making sure to use UTF-8 character encoding. This ensures that special non-ansi characters (like Chinese characters) can be handled properly. Then delete the new database (the one with missing data) and re-import it again using the UTF-8 encoded export file. Be sure to import using UTF-8 character encoding. Also, be careful how you transfer the exported file from the old web server to the new web server. Some services and/or programs can change the character encoding without your knowing it. Your best bet is to zip it up on the old server, transfer it, then unzip it on the new server.

If you have other issues or wish to move a WordPress site in another way (say from one domain name to another), feel free to email us or post in the comments section below. Also check out this website for additional information on moving a WordPress site.

USB Host – The Mystery

The Quest Continues

The sky is the color of a television tuned between channels. An ominous fog pervades the land. The water is now a glowing green amorphous creature nipping at your ankles. Viscous yellow clouds float by at regular intervals. I might think they’re from a child’s toy smoke bomb if not for the fact that they’re setting off my Geiger counter. Clouds melt, running, residents of a bizarre watercolor, expelled by the television the sky. The earth cradles rocks, desert plants and dry grasses, all color-coordinated a neutral beige. A surreal sculpture of a human eye, the size of a small house, glistens and gazes upward at the edge of the park. Is it also concerned about the evicted clouds? The silence would be maddening if not for the odors of wildlife, landscaping and the taste of tepid, unsweetened sun tea in my mouth. This quest is for the strong and the bold.

(My apologies to William Gibson and to you, dear readers, but my quest has evoked the preceding. No, I don’t own a Geiger counter. Although I did run across one that you can include in your next project!)

There really is a surreal and amorphous quest ahead of you if you set off to create a USB host in an external device. (I.e., a device that serves as a USB host.)

You see, your computer(s) may well be the only USB host that you own. Each USB “tree” can have only one host – one host to rule them all. (Even your USB hub is not a host – just a conduit, of sorts.) There can be up to 127 devices, or endpoints, connected to it. Okay, fine. I doubt that I’ll ever have even 27 hooked to one machine.

But what happens if you want to, say, plug a web cam into an Arduino? Woe be unto you, wary traveler! You’ll need a USB host somewhere in the mix. There is a USB host shield available. Some source code for it is freely downloadable but it’s still an alpha version – not even fully implemented. If it does what you need, great. But if you want to program the Arduino to be your host, in addition to doing other things, you may run into problems because the USB host is actually implemented in the Arduino. That uses valuable memory.

Isn’t there a chip that I can plug in to be my USB host? It turns out there is. Sort of… they are sold without firmware so you either need to find someone that sells them “pre-programmed”, or you have to upload the firmware. (There are several firmware images that you can download for free.) If you’re building circuits/devices, that will probably be fine with you. It’s fine with me. But – another catch – if the firmware doesn’t do exactly what you want then you have to write a program, much like you have to program the Arduino. Read more

The Death March

In the world of software development, the term “Death March” is used to describe a final and hopeless effort in which developers forgo all that which is not glorious CODE in a furious and desparate attempt to regain lost honor by completing a project that has fallen shamefully behind schedule–or to die trying. Sleep, sustenance and breath itself are secondary to the great project objective and are permitted only as a means to more CODE. This is the way of the death march.

Random Reflections

In physics, the law of reflection states that for smooth surfaces, the angle at which a wave is incident to a surface equals the angle at which it is reflected. The implication here is that reflection is not random but mathematically predictable. In psychology, reflection is the cognitive activity in which a conscious agent actively contemplates a subject or experience. Again, the implication is that reflection is not random but directed. In computer science and engineering random number generators are required for many functions. These random number generators work by performing a complex mathematic operation on a “seed” number. Once the random number generator is seeded, it will produce output that is unpredictable for anyone that does not know the seed. However, if the seed is known, the output is not random at all but mathematically predictable. In quantum mechanics, probability is used to describe phenomena that is too temporal or complex for direct observation. As methods of observation improve, observed phenomena becomes less probabilistic.

So I guess the question is this. If nothing is random, where do all these thoughts come from? For example, I set out to write not the above, but the following.

In competition, losing often provides more opportunity to learn than winning provided that the loser survives the contest. Losing, however, is not the same as giving up.

On FTP

I have been forced to spend a ridiculous amount of time implementing security for the jointsandjams FTP service. For those interested in a little computer science background, FTP (or the File Transfer Protocol) was first published as a standard in 1971. This standard was simply a set of commands used to perform file operations across a network. At the time, there was no such thing as the “Internet” as we know it today. There was only this experimental network of a few computers called the ARPAnet. This eventually evolved into what the Internet is today, but not for over a period of more than 20 years. Naturally, security was not a major concern for the creators of FTP. They were mostly interested in just building something that worked. Needless to say, they were incredibly successful. FTP is still used by millions of servers every day. Not bad for something that was invented nearly forty years ago for machines less sophisticated than the modern calculator. In the rapidly evolving world of technology, this is something akin to a species of dinosaurs that never went extinct. At any rate, FTP has gone through several revisions since it’s initial publication in 1971. The most recent standard was published in 1985, five years after the current internet standard (TCP/IP v4) was published. Unfortunately, there was still little provision for security even in this revision.

To understand what this means with respect to security, you have to know a little bit about the nature of the early computers. The “PC” or personal computer is a relatively modern invention. The first computers were definitely not for personal use. Originally, they were these massively hulking contraptions that filled entire rooms. Today, NIC cards (the things that connect you to the Internet) are about the size of a large cracker. The ARPAnet equivalents were the size of a refrigerator. Imagine needing something the size of a refrigerator to connect to the Internet. Now imagine that there are only three other computers connected to this Internet. That may seem ridiculous by the standards of today, but to the early pioneers it was a spectacular accomplishment that no one thought possible. Anyway, these machines were primarily financed, maintained and operated by universities and research labs. They were considered extremely valuable resources and competition over who got to use them was fierce. So to alleviate some of this demand, the operating systems that ran them were designed so that multiple processes could be run simultaneously. This eventually led to operating systems that supported multiple users and the concept of a login was born long before a PC ever existed.

So in order for FTP to work on the early system that it was designed for, it had to support login and authentication. This provides a small degree of security. Only users with a valid account and password can use the FTP service. The problem, however, is that standard FTP transmits this information (the account name and password) in cleartext. Actually, it transmits everything in cleartext. In other words, anyone who wants to can read it as it passes from one point to the next across nodes in the network. This was not a significant conern when the network consisted of the machines comprising ARPANET, but in a modern Internet environment where there are millions of users, many of them malicious, it is a very significant problem. A sufficiently motivated and malicious individual could steal passwords as they are transmitted over the network. There are many individuals that could take advantage of this with nothing but a few easily obtainable tools. It could be someone in government, someone at the phone company, or someone on the IT staff at your local university, or library, or any other public access point. Such an individual in possession of a stolen password could then masquerade as that user and access the FTP server gaining control of any data and operations available to that user. Moreover, if the password happens to belong to the administrator account, then the individual could destroy or commandeer the entire system. This is not something a good administrator will take chances with.

To fix this, modern FTP servers are forced to encrypt their communication. This can be technically challenging because the FTP protocal itself is not capable of encryption. There are many difficulties here that I won’t describe in detail other than to say that a lot of configuration at a variety of levels is necessary. For example, it is necessary to configure the FTP software (both server and client), the encryption system, as well as any routing/firewall devices providing access to the service due to the fact that most routing/firewall devices inspect traffic as it passes through in order to determine where it should go and how it should be processed. Unfortunately if the traffic is encrypted they have difficulty interpretting what to do with it. It was quite a nightmare getting it all sorted out, but after over a month of struggling with it, I’m proud to announce that it’s finally working!

Goodbye Emacs :(

As of tonight, I have officially migrated from Emacs to Eclipse as my development environment. There is a cool bit of history here. Emacs is the celebrated UNIX text editor written by none other than the famous hacker Richard Stallman. Richard is a bit of a icon in the programming elite. He studied physics first at Harvard and then at MIT where he did a lot of work with AI systems. Unfortunately, due to a knee injury at the end of his first year at MIT he was forced to abandon international folk dancing and become a full time hacker since physics no longer interested him as much. He went on to found the GNU Project and the Free Software Foundation (FSF). In a way this organization paved the way for OpenSource software by providing a set of development tools and the GNU General Public License (GPL). For example Linux is one of many pieces of software that was developed with tools written by the FSF and published using the GPL. The browser you’re probably using (Firefox) is another example. If it weren’t for Stallman, who knows, you’d probably have to pay for even crappy software.

Anyway, old Stallman did all that and wrote Emacs to boot which I’ve been using to write the code for this very web page! But no longer! I now use Eclipse, which is another piece of OpenSource software. The benefit of Eclipse is that it is graphical (Emacs is text, booo!!) and can integrate with a subversion repository. The result of all this is a more streamlined development process so expect to see some interface improvements very soon.

Routing 101

Earlier this week, during lunch, several of us were involved in a conversation about TCP/IP networking.  One of the questions that was asked was, “If IP addresses and MAC addresses both uniquely identify devices on a network, why do we need both?”.  This question was never really answered so I decided to put together a document that explained this and some of the other basic ideas behind TCP/IP networking.  Unfortunately, I don’t fully understand the subject myself so I was hoping that you guys could throw in additional info/questions/corrections.  Hopefully we’ll all understand it a little more in the end.

I’ll start with the assumption that no NAT is occurring.  NAT was also something that we discussed and I think we all understood it fairly well, but I’ll recap just to be certain.  All IP packets contain a source address and a destination address.  The source address identifies where the IP packet came from and the destination address determines where it’s going.  NAT (Network Address Translation) is a process that rewrites the source address of an IP packet as it travels to its destination. NAT is commonly used in private networks which use non-routable IP address (ie. 10.0.0.0 – 10.255.255.255, 172.16.0.0 – 17.31.255.255, 192.168.0.0 – 192.168.255.255).  Any IP address in this range is not guaranteed to be unique and cannot be routed over the internet.  In other words, these IP address can only be routed over a LAN. So in order for a computer on a local private network to send an IP packet to another computer across the internet, NAT is needed.  NAT rewrites the private (ex. 192.168.0.100) IP address with a unique public IP address and sends it on its way.

NAT is closely related to the function of routers and routers are what run the internet.  In fact, routers are so important you might even say that routers ARE the internet, but that would be a gross simplification.  So what are routers then?  Routers are simply devices that connect one or more networks.  If a network device needs to send information to another network device that isn’t on the local network, it sends it to the network router instead—sometimes known as the default gateway.  It works very much like your mailbox.  If you need to deliver a package to your local neighbor, you just walk on over and knock on the door.  However, if you need to deliver a package to someone 500 miles away, then you write an address on the package and put it in the mailbox and the mail system does the rest for you.  Routers are a bit more complicated than that but the idea is the same.  When a router gets a packet of information, it looks at the destination address and decides where to send it next.  Routers maintain a list of the other networks that they are connected to.  This list is called the “routing table”.  Packets will be routed to different networks based on the destination IP address where other routers will process them.  In this way, packets will be routed from one network to the next until they reach their final destination.

A common question at this point is, “How do routers know the best route to use?”.  The answer is they don’t.  They are stupid.  They blindly look at the routing table and do whatever it tells them to do.  At least that’s how it worked in the early days of networking.  Network administrators manually configured the routing tables based on their personal knowledge of the network structure.  Nowadays routers are actually pretty smart and automatically build their routing table dynamically by communicating with their immediate neighbors.  It’s just like a real neighborhood. If your neighbor sees you burying a corpse in your back yard, eventually the neighbor of your neighbor’s neighbor will hear about it. Routers are they same way.  They are all gossiping little bastards.

So that’s how IP addresses are used to transport data from one location to another, but how do MAC addresses come into play? Well MAC addresses have actually been in play all along but since they operate at a lower level we tend to ignore them.  A MAC address uniquely identifies a Network Interface.  A Network Interface typically takes the form of a card and is also commonly referred to as a NIC (Network Interface Card). A NIC is a device that sends and receives data on a network.  Each NIC is assigned a MAC address by the manufacturer at the factory. Without a NIC, no communication can occur.  NIC’s don’t really know anything about IP addresses.  TCP/IP processing occurs somewhere at the operating system level where everything is pretty and logical.  An application can simply specify an IP address and assume that the data will be delivered.  It doesn’t care about the physical topology that exists between the source and destination addresses.  Things are very different at the hardware level.  At the physical level, a network is just a bunch of NIC’s all wired together.  When a NIC dumps electrical signals onto “the wire”, there may be 50 other NIC’s that are electrically connected to that same wire and all of them will hear and process those signals—also called frames.  These frames contain source and destination MAC addresses, just like IP packets contain source and destination IP addresses.  When an IP packet is handed to the NIC for transmission, it is wrapped up in a frame.  This frame allows the IP address to get from one physical point to the next.  It works like this.  Computer A wants to send a IP packet to Computer B.  Computer A first creates an IP packet and addresses it with the IP address of Computer B and then hands it to the NIC.  The NIC takes the IP packet and says WTFMAN™, I don’t know what to do with this.  So the NIC uses something called ARP (Address Resolution Protocol) to query the network.  Basically, it sends a broadcast message to all nodes on the immediate segment (those that are all electrically connected to the same wire) saying, if you have the IP of Computer B, send me your MAC address.  Computer B will respond and then Computer A will wrap the original IP address up in a frame addressed to Computer B and dump it onto the wire where Computer B will read it. In this scenario, there were no intermediate points between Computer A and Computer B and so only one ARP lookup was necessary.  In many cases there will be multiple intermediate points between source and destination, but the basic process is the same.  A NIC will grab the frame and pass it up to the TCP/IP layer which will decide where it needs to be routed next.  The NIC will then figure out the MAC address of the next stop on the route, using ARP, and then transmit it.  So IP packets are actually getting wrapped and unwrapped in network frames as they travel from point to point across the internet.  It’s kind of like a bad Christmas gift.  You wrap it up and give it to someone who unwraps it and says, “not for me!”  That person then wraps it up again and gives it to someone else.  This process repeats until the present finds it’s perfect match! It’s so BEAUTIFUL!!!

Return top

Info

Section9 is a computer club based out of the Springfield Missouri area. For more information, please see the About Us page or find us on Facebook.