olafalo

DIY Thermostat with a Raspberry Pi

2019-08-25T00:00:00+00:00

I made a thermostat using a Raspberry Pi. It cost about $100, works great, and looks awful. I learned a lot and bothered lots of people by talking about it too much.

But Why?

My apartment is nice, but there’s something I don’t like about it: the AC and heat is loud (exposed ductwork isn’t great for noise). On top of that, the thermostat turns on and off frequently, with cycles that are generally about 10 minutes. This means that watching a movie in the summer is an exercise in frustration, requiring volume adjustments about every 5 minutes.

I looked for a way to change the temperature tolerance of the thermostat, because I don’t really need the temperature to always be within +/-0.5°F of the set temperature, but that wasn’t adjustable. I thought about buying a smart thermostat, but they’re expensive.

Thus, I entered a train of thought that began with “what if I just did it myself?” and ended with, well, doing it myself.

Research Phase

I started with the assumption that I could use a Raspberry Pi as the main controller for my thermostat. They’re cheap (less than $50, including accessories), the newer models have WiFi built-in, and they have super handy GPIO pins that are perfect for interacting with other electronics. So, the plan was to have a Raspberry Pi, with a web interface for control, doing… something that would allow thermostat-like behavior.

The first question is: how will the Pi know what the temperature is? I had a few DHT11 sensors lying around, but they have an accuracy of 2°C (3.6°F), which isn’t really accurate enough for use in a thermostat. The DHT22, on the other hand, has an accuracy of 0.5°C (0.9°F), which is probably fine. I also considered the SCD30, which measures temperature, humidity, and CO₂ levels (cool, right?) but is much more costly. I ended up going with the DHT22.

The next question is: how do we control a thermostat from a Pi? It turns out, it’s fairly simple. There’s a good, in-depth explanation here, but here’s a summary: there are a bunch of control wires that go from the HVAC unit to your wall thermostat. The simplest unit—a heater—has two wires, R and W. If you connect the R wire to the W wire, the heat turns on. If the unit has AC as well as heat, it will also have a Y wire, and connecting the R wire to the Y wire turns the AC on. It’s basically just “connect this wire and that wire,” and which wires to connect depends on your HVAC unit. This means relays are the way to go for control (the reason you can hear your thermostat “click” on).

To find out how to interact with the control wires, I did some prodding with a multimeter on the thermostat that I already had. It turns out that my particular system is a bit odd; it doesn’t have a heating wire connection (W), but instead, the only wired connections were AUX (auxiliary heating), Y (compressor), G (fan), O (heat pump reversal valve), R (24VAC), and C (power). I didn’t need auxiliary heating, and C is for constant power (24VAC), which I didn’t need either because it was easier to just plug the Pi into the wall to power it. In the end, I found what I needed to know: turning the AC on means connecting the R, Y, G, and O wires together. Turning the heat on means connecting the R, Y, and G wires (but not O). Turning on only the fan is done by connecting R and G. In short, turn on the compressor and blower fan to get heat. Also turn on the heat pump reversal valve if you want AC instead.

Armed with knowledge about what sort of switching I’d need to do, I looked around for relays, and ended up using a SainSmart 4-channel relay module ($10). This can be controlled directly from the Pi’s GPIO pins. It’s active low, so connecting an output pin from the Pi to the relay module would switch on the relay if the output pin was low, and switch it off if it was high. Slightly confusing, but that’s fine.

Assembly

I bought the parts I needed:

Raspberry Pi Model 3 B+ ($35)
5V 2.5A MicroUSB wall wart ($7.50)
Any decent SD card ($10)
DHT22/AM2302 ($10 for 2-pack)
4-channel relay module ($10)
Breadboard, jumper wires, multimeter, misc. junk from Sparkfun ($20-30)
Pro Trinket 5V 16MHz ($10)
Total: roughly $100

At first, I attempted to read values from the temperature sensor directly from the Pi. Unfortunately, I had no luck with this. Reading from these sensors requires microsecond-precision timing, and since Linux is not a real-time OS, this was too error-prone to be reliable. Instead, I bought a $10 Pro Trinket, which is a tiny and cheap Arduino board. It’s overkill for what it actually does, but it works. The Trinket is connected to the DHT22, continually reads temperature/humidity values, and writes them over serial (UART) to the Pi.

The Pi, therefore, only needed a couple of connections:

5V and GND connections to the Trinket and the relay module
TX/RX UART connections to the Trinket
GPIO connections to the relay board inputs (GPIO pins 4, 17, and 27)

I also used 18AWG solid copper wire to connect the relays to the wall wiring harness. I chose 18AWG because the pins on the existing thermostat were about 1mm in diameter, so 18AWG would be the best fit to the wall socket. I connected all the common terminals of the relay together to the R wire socket. Then, I connected the Y, G, and O sockets to the normally open (NO) relay terminals 1, 2, and 3, respectively.

After some breadboarding (and plenty of trial and error), the hardware side of things is ready (if not especially pretty)!

Software

You can see the complete project here (please forgive the lack of polish).

The thermostat needs some sort of control interface, and I opted for a web interface. It’s easier and cheaper than wiring up buttons and a screen, plus, sometimes I don’t want to get up to change the temperature. For this, I reached for a simple Go webserver. This project has one dependency: Chart.js, which I use for showing a graph of temperature and humidity over the past 24 hours or so. Everything else is Go’s standard library and regular old HTML/CSS/JS. Besides the nice graph, there’s some basic control options: minimum and maximum temperature, an option to always have the fan on, an enable/disable option, and a setting for the minimum time between switching state (useful for testing).

There are actually three servers that make up the thermostat software. One is the switch controller, which exposes a REST API for setting the HVAC state (e.g. POST/cool to turn the AC on). Another is the sensor server, which returns the latest sensor data as JSON. The last and biggest one is the management server, which serves up the web interface, controls settings, and decides when to turn the AC/heat/fan on and off.

This microservice-ish approach might be overkill, but it was extremely useful in testing—I could work on the management interface from my desktop, while still being able to read actual sensor data and control actual switches. Additionally, it leaves the option for separate components. It would be pretty easy to set up multiple temperature sensors in different rooms, or run the management server on a device that isn’t stuck to the wall.

Software also makes it possible to compensate for the low-ish resolution of the temperature sensor (0.1°C/0.18°F). If the actual temperature is between 22.1 and 22.2°C, for example, the sensor will alternate between reading as 22.1 and 22.2. Doing some basic averaging over time allows us to get a more accurate measure. You could also use humidity to adjust the target temperature, or get weather data from the internet and do… something… based on it. The possibilities are endless.

Next Steps

This thermostat is up and running, and working well! The graph is pretty, the controls are simple and easy. But there’s obviously room for improvement, especially in terms of appearance. If I wanted to really step it up, I would switch to a Raspberry Pi Zero instead, get a custom PCB printed, and package the whole thing up in a 3D-printed enclosure. It would probably look great! But it would also be a lot of work.

This was a fun project, and I like my thermostat a lot more than I would like a normal smart thermostat. It was cheaper, too—comparable with a regular dumb thermostat, if you don’t count the time spent. The jury is still out on whether or not it will be a good conversation starter, however.

An Image Host of Your Own, Part 1

2018-11-15T14:36:24+00:00

The result of this post is live and running at img.olafalo.net. Check it out! (Please excuse my front-end.)

Intro

A few years ago, starting an image host was a daunting task. The first few users might not have been too hard, but scaling was a problem. How do you reliably serve images to millions of users?

The answer these days is, of course, to let someone else do it. AWS S3 is brilliant for this. You can generally stuff whatever you want into an S3 bucket, and anyone can reliably access it. Add on tools like CloudFront, and now anyone can reliably access it quickly.

In this post, we’ll create a backend for the simplest MVP image host possible. All it will do is allow users to upload an image, then get back a sharable link to that image. The cool part, though, is that this setup could theoretically scale to millions of global users at any time without breaking a sweat, because all the technology we’ll be using (mostly S3, Lambda, and CloudFront) scales very well.

Step 0: Setup

If you want to follow along, you’ll need an AWS account. You’ll probably also want to set up the AWS CLI tool. Some background knowledge would be helpful too, but I’ll explain the basics as we go.

Step 1: S3

I’m going to create 3 S3 buckets:

Deployments: A private bucket which we use for lambda deployments, more on that later
StaticWebsite: A public bucket that’s set up as a website, to host the static part of our website (index.html and such)
ImageBucket: The bucket that the actual images will go into

I make an effort to use CloudFormation for everything, because “make an image host without writing any code!” sounds much better than it actually is, and I didn’t want to spend an entire day taking screenshots of the AWS console. Here’s basestack.yaml, which describes a CFN stack with those three buckets:

Resources:
  Deployments:
    Type: AWS::S3::Bucket
    Properties:
      AccessControl: Private
  StaticWebsite:
    Type: AWS::S3::Bucket
    Properties:
      AccessControl: PublicRead
  ImageBucket:
    Type: AWS::S3::Bucket
    Properties:
      AccessControl: PublicRead

To deploy this stack, we can use the aws CLI tool:

aws cloudformation deploy template-file basestack.yaml --stack-name BaseStack --capabilities CAPABILITY_IAM

The --capabilities flag may not be necessary in this case, but you generally need it when you’re changing permissions. The docs have this to say:

Some stack templates might include resources that can affect permissions in your AWS account, for example, by creating new AWS Identity and Access Management (IAM) users. For those stacks, you must explicitly acknowledge their capabilities by specifying this parameter. The only valid values are CAPABILITY_IAM and CAPABILITY_NAMED_IAM.

If the deployment went well, congrats! You have some S3 buckets. I recommend creating a shell script or two for building and deploying, once we have something to build.

Step 2: “Serverless” Just Means Someone Else’s Servers

To allow image uploads, we’re going to create a lambda function that has upload access to our ImageBucket. We’re also going to create an API Gateway endpoint that allows anyone to send image information to our lambda.

Lambdas can be written in several languages, but I went with Go for this one. I’m a big fan of Go; it’s quick, predictable, and there’s no need to sprinkle async keywords everywhere or to put your entire lambda in one long promise chain. Anyway, here’s upload.go:

package main

var awsSession *session.Session

// Get the name of the upload bucket as a config var
var bucketName = os.Getenv("UPLOAD_BUCKET")

// getSession returns a new session if one did not already exist, and the
// existing session if it had already been created.
func getSession() *session.Session {
	if awsSession != nil {
		return awsSession
	}
	// Initialize a session that the SDK will use to load configuration,
	// credentials, and region from the shared config file. (~/.aws/config).
	awsSession := session.Must(session.NewSessionWithOptions(session.Options{
		SharedConfigState: session.SharedConfigEnable,
	}))
	return awsSession
}

// Lambda input structure
type imageEvent struct {
	Image64   string // image contents in base64
	Extension string // file extension, e.g. "png"
}

// Lambda output structure
type uploadResult struct {
	URL string
}

// The lambda function itself
func uploadImage(ctx context.Context, event imageEvent) (uploadResult, error) {
	uploader := s3manager.NewUploader(getSession())
	image, err := base64.StdEncoding.DecodeString(event.Image64)
	if err != nil {
		return uploadResult{}, err
	}
	// Create a key for the image to store in S3, a UUID with an extension
	imageID := uuid.NewV4().String()
	imageKey := fmt.Sprintf("i/%s.%s", imageID, event.Extension)
	imageReader := bytes.NewReader(image)
	uploadParams := &s3manager.UploadInput{
		Bucket: &bucketName,
		Key:    &imageKey,
		Body:   imageReader,
		// I know, I know. We could definitely find a better way to set the
		// content-type, but this works well enough as an MVP.
		ContentType: aws.String(fmt.Sprintf("image/%s", event.Extension)),
	}
	// Do the upload and return its URL
	_, err = uploader.Upload(uploadParams)
	if err != nil {
		return uploadResult{}, err
	}
	return uploadResult{URL: fmt.Sprintf("%s", imageKey)}, nil
}

func main() {
	lambda.Start(uploadImage)
}

I saved this file as upload.go in a directory called image-upload. To build this for use in a lambda, we have to do something like this:

mkdir -p tmp/bin
GOOS=linux GOARCH=amd64 go build -o tmp/bin/image-upload ./image-upload
mkdir -p tmp/dist
zip -j tmp/dist/image-upload.zip tmp/bin/image-upload

This gives us tmp/dist/image-upload.zip, which we can use as the code for the lambda.

Now we need another CloudFormation stack, which will contain the lambda and the API Gateway endpoint. The resources we need are:

The lambda itself
A role for the lambda that allows uploading to S3
A permission that allows API Gateway to invoke the lambda
An API Gateway REST API
A POST method for that API (the upload endpoint)
An OPTIONS method for the API (so that we don’t run into CORS issues)
An API Gateway deployment for our API

Yikes. We also need a few parameters, so that we know the names of our S3 buckets, and so that we can have a deployment prefix that updates each time we deploy. Putting it all together, this is what we get, stack.yaml:

Resources:
  S3UploadRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          -
            Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - sts:AssumeRole
      Path: /
      Policies:
        -
          PolicyName: S3UploadPolicy
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              -
                Effect: Allow
                Action: s3:*
                Resource: "*"
  ImageUploadLambda:
    Type: AWS::Lambda::Function
    Properties:
      Handler: image-upload
      Runtime: go1.x
      Environment:
        Variables:
          UPLOAD_BUCKET: !Ref ImageBucket
      Role: !GetAtt S3UploadRole.Arn
      Code:
        S3Bucket: !Ref DeploymentsBucket
        S3Key:
          Fn::Join:
            - "/"
            -
              - !Ref DeployPrefix
              - image-upload.zip
  ImageUploadLambdaPermission:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !Ref ImageUploadLambda
      Principal: apigateway.amazonaws.com
      SourceArn: !Sub "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${UploadAPI}/*/POST/*"
    DependsOn:
      - ImageUploadLambda
      - UploadAPI
  UploadAPI:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: UploadAPI
      BinaryMediaTypes:
        - image/jpeg
        - image/png
        - image/gif
        - image/webp
      EndpointConfiguration:
        Types:
          - EDGE
  UploadAPIPost:
    Type: AWS::ApiGateway::Method
    Properties:
      AuthorizationType: NONE
      HttpMethod: POST
      ResourceId:
        Fn::GetAtt:
          - UploadAPI
          - RootResourceId
      RestApiId: !Ref UploadAPI
      MethodResponses:
          -
            StatusCode: 200
            ResponseModels:
              application/json: 'Empty'
            ResponseParameters:
                method.response.header.Access-Control-Allow-Headers: false
                method.response.header.Access-Control-Allow-Methods: false
                method.response.header.Access-Control-Allow-Origin: false
          -
            StatusCode: 500
            ResponseModels:
              application/json: 'Empty'
            ResponseParameters:
                method.response.header.Access-Control-Allow-Headers: false
                method.response.header.Access-Control-Allow-Methods: false
                method.response.header.Access-Control-Allow-Origin: false
      Integration:
        Type: AWS
        IntegrationHttpMethod: POST
        IntegrationResponses:
          -
            StatusCode: 200
            ResponseParameters:
              method.response.header.Access-Control-Allow-Headers: "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token'"
              method.response.header.Access-Control-Allow-Methods: "'POST,OPTIONS'"
              method.response.header.Access-Control-Allow-Origin: "'*'"
          -
            StatusCode: 500
            SelectionPattern: .+
            ResponseParameters:
              method.response.header.Access-Control-Allow-Headers: "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token'"
              method.response.header.Access-Control-Allow-Methods: "'POST,OPTIONS'"
              method.response.header.Access-Control-Allow-Origin: "'*'"
        Uri:
          Fn::Join:
            - ""
            -
              - "arn:aws:apigateway:"
              - !Ref AWS::Region
              - ":lambda:path/2015-03-31/functions/"
              - !GetAtt ImageUploadLambda.Arn
              - "/invocations"
  UploadAPIOptions:
    Type: AWS::ApiGateway::Method
    Properties:
      AuthorizationType: NONE
      RestApiId: !Ref UploadAPI
      ResourceId: !GetAtt UploadAPI.RootResourceId
      HttpMethod: OPTIONS
      Integration:
        IntegrationResponses:
        - StatusCode: 200
          ResponseParameters:
            method.response.header.Access-Control-Allow-Headers: "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token'"
            method.response.header.Access-Control-Allow-Methods: "'POST,OPTIONS'"
            method.response.header.Access-Control-Allow-Origin: "'*'"
          ResponseTemplates:
            application/json: ''
        PassthroughBehavior: WHEN_NO_MATCH
        RequestTemplates:
          application/json: '{"statusCode": 200}'
        Type: MOCK
      MethodResponses:
      - StatusCode: 200
        ResponseModels:
          application/json: 'Empty'
        ResponseParameters:
            method.response.header.Access-Control-Allow-Headers: false
            method.response.header.Access-Control-Allow-Methods: false
            method.response.header.Access-Control-Allow-Origin: false
  UploadAPIDeployment:
    Type: AWS::ApiGateway::Deployment
    Properties:
      RestApiId: !Ref UploadAPI
      StageName: v0
    DependsOn: UploadAPIPost
Parameters:
  DeployPrefix:
    Type: String
    Default: "0"
  DeploymentsBucket:
    Type: String
  ImageBucket:
    Type: String

I’ve never heard anyone call CloudFormation “terse”.

Anyway, here’s my deploy.sh that I used to deploy the whole backend in one go:

#!/bin/bash
TEMPLATE_FILE=basestack.yaml
STACK_NAME=BaseStack

function getResourceId {
    aws cloudformation describe-stack-resource \
        --stack-name $STACK_NAME \
        --logical-resource-id $1 \
        | jq --raw-output '.StackResourceDetail.PhysicalResourceId'
}

# create base stack
aws cloudformation deploy \
    --template-file $TEMPLATE_FILE \
    --stack-name $STACK_NAME \
    --capabilities CAPABILITY_IAM

# sync dist folder
DEPLOY_PREFIX=deploy-$(date +%s)
aws s3 sync ./tmp/dist s3://$(getResourceId Deployments)/$DEPLOY_PREFIX --delete

# sync website (once we have one)
# aws s3 sync ./tmp/web s3://$(getResourceId StaticWebsite) --delete

DEPLOY_BUCKET=$(getResourceId Deployments)
IMAGE_BUCKET=$(getResourceId ImageBucket)
TEMPLATE_FILE=stack.yaml
STACK_NAME=ImageHost

aws cloudformation deploy \
    --template-file $TEMPLATE_FILE \
    --stack-name $STACK_NAME \
    --capabilities CAPABILITY_IAM \
    --parameter-overrides DeployPrefix=$DEPLOY_PREFIX \
        DeploymentsBucket=$DEPLOY_BUCKET \
        ImageBucket=$IMAGE_BUCKET

You should be able to deploy the whole backend with one command now, which is neat. We now have an endpoint that you can hit to upload an image and get back a link to it! We can’t actually view the uploaded image, though, because we haven’t set an access policy on the S3 bucket we use for hosting. Let’s set up a CloudFront distribution to get around that (and also get nice CDN junk, as well as SSL and a custom domain).

Step 3: Distribute Content (with a Network)

We could make our CloudFront distribution with CloudFormation, but we’ve had plenty of CFN already, and the console is pretty easy for this part. Go to CloudFront in the AWS web console, and click the friendly blue “Create Distribution” button.

We want to create a web distribution. We want an Origin Domain Name of the static website bucket (we’ll make another distribution for the image hosting bucket later). We do want to Restrict Bucket Access, Create a New Identity for the Origin Access Identity, and Yes, Update Bucket Policy. This will allow us to access the contents of the bucket through the CDN. The Default Root Object should probably be index.html. Maybe you also want to force HTTPS, but that’s optional. Everything else should be good by default.

Once that’s done, we can edit the distribution we just created. We should create another origin for the distribution, which points to the image hosting bucket (do the same restriction of bucket access, create another new identity, and let CloudFront update the bucket policy for you). We should also create a new Behavior, with a path pattern of i/*, with an origin of the image hosting bucket.

Once all that is done, we should have our subdomain ([keyboard-mashing].cloudfront.net). This will host all content from our static website bucket, unless the path starts with i/*, in which case it will host content from our images bucket. Here’s an example of the flow of things:

Browser requests [whatever].cloudfront.net
CloudFront returns our hosted index.html as well as any other hosted files
User uses our web frontend to POST image data to our lambda, which gets uploaded to the image host S3 bucket
Lambda returns a link to [whatever].cloudfront.net/i/something.png
CloudFront gets requests for [whatever].cloudfront.net/i/something.png and serves up the requested image, from the image host bucket (because the path starts with i/*).

Step 4: Congratulations

We’re done with the backend! We now have infrastructure to host a static website and any number of images, as well as an upload API. Next, we need the web front-end that will allow people to actually use our host.

For extra credit, use CloudFront to allow your API to be accessed from the same .cloudfront.net domain. This will remove the requirement for an OPTIONS endpoint, because same-domain requests (exact domain matches, to be clear) don’t need to be pre-flighted.