The Issue

I had made some small test projects, with buttons that seemed to work fine. As I worked on a more substantial project, it came time to create a menu for the user, and suddenly, things went off the rails. The issue? No UI component would accept mouse clicks.

The Bad Assumptions

Now, I made lots of incorrect assumptions. This happened because of:

• Leaning on previous experience thinking I knew what was up.

• Read the docs. Unless I misunderstood, the code seemed to check out per examples there.

• History. I remember smaller projects, where a button click would do something.

• Deep diving on the Internet. Lots of similar issues or even the same issue and not fully understanding the root causes.

• Thought the click coordinates were not in the right spot (since the elements did not react, like hovering over a button didn’t change it, even though it was correct in the scene in terms of setup).

The Bad Code

Now, the code itself was correct per se. I lost a lot of time trying to debug, including adding code simply to confirm via _input() where I was clicking. Essentially all combinations of debugging attempts confirmed activity was happening but no events were being processed by the UI components, regardless of component.

Fail Attempt One

All UI elements are based on the Control Node type. It contains a property in the config called mouse_filter, found in the Mouse→Filter section of the Inspector. (More information on GUI controls can be found here.). This mouse_filter setting, which is stop, pass, and ignore, dictates what the node should do. In Godot, for a button to handle a press event from the mouse, it should set this value to stop. However, all nodes above it in the tree in your scene, should have their mouse_filter set to ignore.

To quickly summarize the mouse_filter options:

Stop: The node will consume the event.

Pass: The node will consume the event and pass it to the parent node to consume.

Ignore: The node discards the event.

Even knowing this, and confirming many times the settings were correct, it did not work.

Fail Attempt Two

After reaching deeper into Godot history, I was thinking maybe changes related to how signal processing was handled is causing me an issue. Maybe I forgot a step in terms of config, as in Project→Project Settings, Input Map. I actually had keyboard events working, and they never failed to work for me. As I was using the function _input(), I thought maybe mouse clicks were supposed to be done in a similar way. I never fully went down this path, but it did lead to more searching.

I even found references to events failing or being inconsistent, including sometimes on Mac silicon based Macs. However, that seemed extreme (and often from older posts, before Godot 4.0 was even out).

However, after much resetting signals, changing event code, testing input maps, still no progress.

Break Time

Not totally. I decided to go back to new projects and try again, and as expected, it worked. So it brought my thinking right back to something that must be wrong in this specific project.

Third Attempt is the Charm

Finally, hope! This project I have worked on requires transitioning between scenes. A TransitionManager was created earlier in the project to fade between scenes. This will matter in a moment.

While randomly searching I found a post about one small feature of the Godot debugger. There is a panel named Misc, which happens to show the node that processed the most recent event. I was not aware of this. Immediately I checked it out. Sure enough, while running just a single scene, not the entire project, it referenced a component in a different scene. You guessed it, the TransitionManager.

The TransitionManager has a ColorRect, which is a Control node type. I blindly assumed a random color rectangle would never block events. In fact, I never thought about another scene being the issue at all. What was learned here is, when you autoload a scene (which is in Project→Project Settings, Globals, Autoload), these scenes last for the duration of the project runtime. By design, visual components generally have mouse_filter set to stop, to avoid causing downstream issues in your scene. I, like many, presumed most would be set to pass or ignore. I was wrong.

The Debugger Misc tab clearly showed the ColorRect was stopping the pressed() event. I jumped over to the TransitionManager scene, went to the ColorRect Mouse→Filter properties and set it to ignore. Restarted the scene, everything works just as expected.

Summary

While it took more time then I would have liked, I did learn quite a few things along the way, as well as got much better understanding and moving around the Godot editor. To summarize the issue, if click events are not working as you expect:

• Confirm the immediate scene tree Control-based nodes have the correct mouse_filter settings.

• Double check your signal is configured to the correct script location and function.

• Running the problematic scene, check inside the Misc debugger panel to see which node consumed the click event.

Additional Debugging

One area that didn’t really help fix the issue did give me insight into event processing was logging. For example, in your scenes script:

func _input(event) -> void:	
	if (event.as_text() == "Left Mouse Button"):
		print(event.as_text())
		print(event.get_property_list())

Events sent by the OS, such as a mouse click, can be captured in the _input(event) function. You would use this function only to capture the event at the time it happened. I used this for example to get coordinates, etc to see what might be wrong. Meaning more directly, this function gave me output!

While informative, it sent me down a very incorrect path in terms of the issue. There is a lot to processing input, whether a one event item like clicking a menu button, to holding down multiple keys at the same time. You can find more information on which functions to use and when in the official docs here.

Installation

You can directly grab the installer either on Godot’s website at:

https://godotengine.org/download/macos/

You can install view Homebrew but I was wanting to go through their standard process, just to see what gotchas I might hit. Just download the zip and run the executable.

As for obtaining GitHub Desktop, you can download here:

https://desktop.github.com/download/

Be sure to grab the correct version (in my case the Apple silicon download).

The Homebrew recipes for both these applications are as follows:

# Godot
brew install --cask godot

# GitHub, not to be confused with the "git" download
brew install --cask github

GitHub Desktop Setup

After you install the GitHub Desktop application, launch it via Applications or Spotlight. If you do not have a GitHub account, now would be the time to set it up over at:

https://github.com/

If you are already logged into Github, it will redirect you to authorize the GitHub Desktop app:

From this point, you will need to configure Git locally, which is done via the following screen.

Once you add your details and click finish, you will be greeted with the quick start screen:

Godot Project Setup

As this is not a tutorial on how to use Godot (the projects 2D tutorial is a great place to start here), the basic steps to setup Godot with GitHub Desktop (and Git) is:

• Create your project OR launch an existing project in Godot. We will create a new one.

• Add your Project Name, correct path. You can ignore the "Version Control Metadata" setting for now and leave it as "Git". We talk more on this later.

• Click Create & Edit. You will get a default empty 3D scene:

Git Repository Creation

• In the GitHub Desktop application, click "Create a New Repository on your Local Drive".

• Fill in the details:

  • Name: Your project name.

    • ALERT: Due to the way this works, it will create a nested folder inside your project. Be sure to use the same name as the top level folder, and remove the duplicate name in the file path before clicking "Create Repository".

  • Description: For your repo description.

  • Local Path: Where you have your project located, from the top folder of the project.

  • Click "Initialize this repository with a README", you can add updated content later.

  • In Git Ignore, find Godot and select it.

  • If you care to add a license, select on here as well under License. Godot is released under the MIT license. You can find more project license details here.

  • Click Create Repository.

Here is a screenshot of my setup:

I left the misconfigured path above just to drive home the reminder to remove the project folder name from the Local Path before you click "Create Repository". If you do not, it will create a folder with your options inside your root which you do not want!

Now presuming all is good, you can click the History tab to see the initial project files and new git files already in your default "Initial commit". The default screen will state "No local changes" because you have not done anything new since.

Now click on "Public repository", and select your options as desired. I will keep this private for now in this example:

From here you can click the "View on GitHub" button to see your new repo:

Why?

Why did we even need to go through this? Currently (as of late 2024) there is a Git plugin integration. In fact we saw a mention of it earlier during the Godot install. However, despite the efforts of the Godot Git plugin developers, it remains problematic, especially as Godot itself evolves. Eventually it will get there. (The reason for the dropdown itself was to allow other source control tools to be options in the future).

You could also do all this via the command line if you are comfortable with it, or prefer it. Given the visual nature of this tool, I opted to use the desktop option simply so I can easily see as I create branches and commits as the game progresses.

Summary

From here, carry on like normal. Your source code strategy is up to you. Just be sure now after you locally test your code and it seems good, be sure to check it in locally to your origin on GitHub!

Prerequisites

The following guide is Mac (silicon) based. However most steps should remain nearly the same between systems.

To get started:

• Log into your Supabase account.

• Navigate to your project, then to Project Settings→ Configuration (Database) → Connection String (psql). You will be using configuration data here later.

• Find your project’s database password (will be needed later).

Installing Basic Postgres Tooling

We do not need a full instance of Postgres to perform the backup, but we do need the core tooling. This is available via the libpq library. You can find more information on this library here.

First, update and optionally upgrade homebrew:

brew update
brew upgrade

Once this completes, use homebrew to install libpq:

brew install libpq

Once finished, you will need to update your environment because brew does not install these tools in any regular location (keg-only).

Might Work

One option is to use symlink via:

brew link --force libpq

However, for myself, this did not work, so to undo this, drop the force flag and change the subcommand to unlink:

brew unlink libpq

If I sort this out later I will update.

Did Work

What DID work however is the following (which is part of the final output from brew):

 echo 'export PATH="/opt/homebrew/opt/libpq/bin:$PATH"' >> ~/.zshrc

Then update your terminal environment via:

source ~/.zshrc

From here, you can test that you can reach the various tools by entering pg_dump, pg_restore, etc.

Backing Up Your Projects Database

Now that the tooling is present, we need to build the correct connection string for the various tools. For this post, we are focusing only on creating the local backup.

In your Supabase projects settings you navigated towards earlier, you will see a screen like this:

The provided connection string for PSQL is nearly exactly what you need. We simply need to update the command and tweak any options needed. Given the above (minus the id), here are three backup options. I navigated to ~/Downloads before running these commands:

Schema and Data

pg_dump -h aws-0-us-west-1.pooler.supabase.com -p 6543 -d postgres -U postgres.<your_project_id> supabase_dump_backup.sql

Schema Only

pg_dump -h aws-0-us-west-1.pooler.supabase.com -p 6543 -d postgres -U postgres.<your_project_id> --schema-only > supabase_schema_backup.sql

Data Only

pg_dump -h aws-0-us-west-1.pooler.supabase.com -p 6543 -d postgres -U postgres.<your_project_id> --data-only -Fc > supabase_data_backup.dump

Alternatively, to get an uncompressed readable version, drop the -Fc flag. That flag is to help provide a compressed and more convenient version for later usage in pg_restore.

Do be mindful if you have customer/real data in these backups and keep them in a safe place and also delete if unneeded.

What is Expo Go?

Expo Go is a sandbox for working with the Expo development environments. Think of it as a tool that will allow you to stand up your React Native app using Expo tooling, and then via Expo Go, deploy the mobile app to your device (Android or iOS).

Expo Go is not intended for production app deployment. Your app codebase however is the same. This means, after you iterate on your app, you can take the codebase and then move into the full Expo environment to eventually get your app in the Play or App stores. This is done by converting your Expo Go project to an EAS (Expo Application Services) Build style project pipeline, which allows cloud or local builds of your application, shared testing links, app store submission, etc. Expo Go is free, but Expo EAS does cost money after a free tier. You can iterate on the bulk of your project development before launching such a build.

Prerequisites

It’s best to start by reading the quick start primer for the various things we will be doing here. The immediate one to focus on is React Native to get a sense how it all ties together:

React Native Development Environment Setup Quick Start

For this tutorial:

• You need to be on a Mac.

• Have Node installed (one method to do so here is Homebrew).

• Your Android device will be on the same wifi network as your Mac.

Initial Steps

Expo Go Installation (on Mac)

Via npx, we will use the create-expo-app generator to create a mobile app called ExpoSample. From the command line in your preferred folder:

npx create-expo-app ExpoSample

You may be prompted to install an updated generator, if so, proceed:

rob@Robs-MacBook-Pro-M2-MAX react-native-test % npx create-expo-app ExpoSample
Need to install the following packages:
create-expo-app@2.3.5
Ok to proceed? (y) y

From here, change into the directory and open Visual Studio Code.

rob@Robs-MacBook-Pro-M2-MAX react-native-test %  cd ExpoSample 
rob@Robs-MacBook-Pro-M2-MAX ExpoSample % code .

To smoke test Expo Go is installed and ready to go run:

rob@Robs-MacBook-Pro-M2-MAX ExpoSample % npx expo start

The output will look something like this:

Expo Go Installation (on Android)

There are several ways to run React Native/Expo applications. In this case we want to quickly get the sample application to a real Android device. To do this, first we must install Expo Go on the Android device.

• Navigate to the Expo homepage.

• Click Install Expo Go for Android, currently the Expo Go SDK is version 51:

Via manual installation, scan the QR code from the dialog that opens:

Note: We are not using Expo Orbit in this example.

This QR code will take you to the Expo application in Google Play store. From here, install as you would an app from the Play store.

Once open, using the Expo Go app, scan the QR code that was presented in the terminal earlier and it will download, across your local wifi network, the application from your computer to the Android device.

Modifying ExpoSample

For our sample app named ExpoSample, the root page is located in:

/app/(tabs)/index.tsx

Let’s change the following code:

To something more personal like “Welcome to us!”, just to see the change reflected in the app:

Save the file, then in the terminal press the r key to reload the application bundle to see the change. You will see the “Reloading apps” message:

In the device, Expo Go will relaunch the app bundle with the new text:

That’s it. There are things to consider in this setup.

Limitations

Expo Go is a sandbox. What it gives us is a quick way to start building and running React Native apps by providing a simple method to deploy the app to your phone. More directly, this means in its current setup, you will not be able to run your app outside of Expo Go’s sandbox on your device. This is by design and is intended to help a developer make progress fast. To make your app available before a proper app build and store deployment include:

• Setting up a tunnel to allow access to your running app.

• Share dev versions via Expo and Expo CLI.

• Share pre-release versions via OS based previews.

• Google Play Console or Apple App Store TestFlight.

Expo Go itself does not allow custom native code in its sandbox. Again this is by design. However the Expo SDK is quite mature already, and supports building your own custom modules into Expo SDK itself, which build later into native projects. You can find more on custom native code here. A regular Expo build however does support custom native code because it’s just a regular build app build environment.

The create-expo-app generator is updated quarterly to use the current latest React Native version. More details on this are found here.

Prerequisites

As noted, we will assume:

• Visual Studio Code (VS Code) is installed.

• Docker Desktop is installed.In earlier posts, I have discussed how to setup Visual Studio Code, Python and Docker, which you can see revisit in these previous posts:

Postgres

As we will need a database to call from our Python script, the first thing we will do is grab the Postgres image and launch it locally. Much like the previous MongoDB Docker image, once you launch Docker Desktop:

• In the Search bar, type postgres.

• Click "Pull", to download the latest image. (Tag is set to "latest" by default.)

Once completed it will have a checkmark next to the "latest" tag.

Configure Docker and Run Postgres

This is where things slightly diverge from the earlier MongoDB image configuration. For postgres, we must pass in a default username and password. As Docker Desktop does not support this, we will do this on the command line now that we have the image (all the tools required to work with Docker were downloaded when we set up Docker Desktop in the first place).

• Open a terminal.

• Run "docker -v" to confirm docker is running.

• In the terminal run the following:

docker run --name local-postgres -p 5432:5432 -e POSTGRES_PASSWORD=mypassword -d postgres

The above command will:

• Run a docker container named "local-postgres".

• On port 5432 locally, so our Python script can later find it.

• POSTGRES_PASSWORD, which is the initial master password, will be passed into the environment variables via -e.

• Run detached via -d.

At this time, in the Containers view of Docker Desktop, you will see your running "local-postgres" container:

From this point you can now use Docker Desktop to stop and restart your container.

To test your Postgres instance, let’s create a database.

• Click on your postgres container.

• Click on the Terminal tab.

• Optionally move into the bash shell by typing "bash".

• Log into the running instance of postgres with the default user:

psql -d postgres -U postgres

• Now using DDL lets create the initial database and user so we can later access it in our Python script. Run each line individually:

CREATE DATABASE test_database;
CREATE USER test_user WITH PASSWORD 'test_password'; 
GRANT ALL PRIVILEGES ON DATABASE test_database TO test_user;

Type "exit" to leave psql.

psql

psql is the primary command line tool to use to interact with postgres. If you would like to now connect as the user you just created, your command line in the container terminal will look like:

psql -d postgres -U test_user -W

Then enter your password when prompted.

Once in, you can run the various commands, like list the database via "\list":

Let’s add a table and some data for us to work with via Python later. In psql, we must first connect to the database as the user we are logged in as, so run:

\c test_database

Then run the following DML to create a table for user data:

CREATE TABLE users (
  id SERIAL PRIMARY KEY, 
  firstname VARCHAR(255),
  lastname VARCHAR(255),
  username VARCHAR(255), 
  email VARCHAR(255),
  date_created DATE,
  last_update DATE
);

Confirm the table was created running "\dt+":

Note it was created in the public schema.

Now run the following to insert some rows, each row is one user:

INSERT INTO users (firstname, lastname, username, email, date_created, last_update)
VALUES
  ('John', 'Doe', 'jdoe123', 'jdoe@example.com', '2023-01-01', '2023-01-01'),
  ('Jane', 'Doe', 'janed', 'jane@example.com', '2023-02-01', '2023-02-01');

Now you can query the data via SELECT:

SELECT * FROM users

And the result:

NOTE: If you have an issue with the syntax and get an error like "ERROR: syntax error at or near "SELECT", add a leading semicolon. If you made an error earlier, sometimes it won’t detect the closing semicolon from earlier and think you’re still typing commands. Your query would look like this: ";SELECT * FROM users;".

Looks like we are good to go.

Accessing Postgres via Python

The most common Postgres data adapter for Python is currently Psycopg2, so let’s install it. In VS Code, open a terminal and run:

pip install psycopg2-binary

Why the "-binary"? Because the way psycopg2 is built has some underlying dependencies so we do not want to worry about building it ourselves.

Now in VS Code create a Python file, app.py and paste the following code:

import psycopg2

conn = psycopg2.connect(
    host="localhost",
    database="test_database",
    user="test_user",
    password="test_password",
)

cur = conn.cursor()

cur.execute("SELECT * FROM users;")

records = cur.fetchall()
for row in records:
    print(row)

conn.close()

You can now either run via Run and Debug in VS Code, or in your terminal run "python3 app.py". Adjust your version of Python and the filename accordingly.

Your output will look like this:

rob@Robs-MacBook-Pro-M2-MAX python-postgres % python3 app.py 
(1, 'John', 'Doe', 'jdoe123', 'jdoe@example.com', datetime.date(2023, 1, 1), datetime.date(2023, 1, 1))
(2, 'Jane', 'Doe', 'janed', 'jane@example.com', datetime.date(2023, 2, 1), datetime.date(2023, 2, 1)) 

Summary

That’s it! As with any database, there is a lot to consider:

• Security (authentication, authorization, roles, etc).

• Performance (SQL versus your language of choice like Python).

• RDBMS flavor (example here was Postgres, but other SQL systems will be quite similar like MySQL).

The code for this post can be found on GitHub here.

If you have spent any amount of time on the Internet, you have most likely come across websites that aggregate data from multiple websites and/or APIs. Today’s post is to show just one example of how this activity is accomplished.

Prerequisites

Like most projects, the tools we will be using here are:

• Visual Studio Code.

• Node.js.

• Homebrew.

The current version of these tools should be fine as we will not be doing anything crazy right now. Anything else we need installed we will cover as we go. If you need refreshers on how to install these tools, you can refer to a couple earlier posts.

VS Code and Homebrew Installation

Node.js Installation (via Homebrew)

Basic Application Setup

To get started, fire up VS Code and open a new empty folder in your normal repo folder.

• Open a terminal in VS Code, in your folder, in my case /Users/rob/Repos/simple-scraper.

• To initialize our Node.js project, type:

npm init

We will leave all these values default, so press enter/return until the process is complete. You will now have a package.json file with the generated setup.

• In the root of your folder, create index.js, which is the point of entry of our server.

Now with the fundamental Node.js application structure in place, let’s grab the third party libraries we are going to be using.

cheerio

cheerio is a server-side implementation of core jQuery. We will use this to parse the web pages for information we are interested in (as it allows DOM traversal and manipulation).

NOTE: cheerio does not render a page or run any JavaScript, you need other tools for that.

To install, in the terminal run:

npm install cheerio

Your package.json file (your Node.js application configuration) will now be updated, package-lock.json will be created, and now create node_modules and download the required files.

express

A web framework built for node, that will listen on the network for incoming requests and serves responses. Think of this as our web application/web server.

To install express:

npm install express

axios

A HTTP client for Node.js. It allows us to interact with other APIs behind the scenes. Axios is basically what we will use to fetch data.

To install axios:

npm install axios

Great, now we have the three key library dependencies installed. Think of express as our server, and axios as being used to communicate to other servers.

Application Buildout

Now it’s time to write some code. Open up index.js and let’s get started.

A common convention is to call your server instance (meaning your instance of express), "app". It could be called anything, but this is maybe the most common. (For example, say you had both HTTP and HTTPS services available, you would run something different like app_http and app_https if you wanted to make clear the difference between the servers).

The basic initial code to import the third party libraries and kick off your express server on port 8080 will look something like this:

const SERVER_NAME = "simple-scraper";
const SERVER_PORT = 8080;
const express = require("express");
const axios = require("axios");
const cheerio = require("cheerio");

const app = express();

app.listen(SERVER_PORT, () => {
  console.log(`${SERVER_NAME} running on ${SERVER_PORT}`);
});

Running the Application

Lets now run our application via npm run start:

rob@Robs-MacBook-Pro-M2-MAX simple-scraper % npm run start
npm ERR! Missing script: "start"
npm ERR! 
npm ERR! Did you mean one of these?
npm ERR!     npm star # Mark your favorite packages
npm ERR!     npm stars # View packages marked as favorites
npm ERR! 
npm ERR! To see a list of scripts, run:
npm ERR!   npm run

npm ERR! A complete log of this run can be found in: /Users/rob/.npm/_logs/2023-08-25T20_55_36_897Z-debug-0.log

Disaster! What happened? We did not actually define the command "start". How do we fix this? Open package.json, our application configuration, and lets add a start command in the scripts section, as follows:

  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "start": "nodemon index.js"
  },

Now when start is run, nodemon will launch and monitor our index.js file for any changes and restart the running node process as necessary to keep it updated. Lets run "npm run start" again:

rob@Robs-MacBook-Pro-M2-MAX simple-scraper % npm run start        

> simple-scraper@1.0.0 start
> nodemon index.js

sh: nodemon: command not found

I left this until now to show how to debug aspects of your development environment. What this is saying is, we need to install nodemon. To do so run:

npm i nodemon

That’s it (i is shorthand for install). Now we can successfully launch our application via "npm run start":

rob@Robs-MacBook-Pro-M2-MAX simple-scraper % npm run start

> simple-scraper@1.0.0 start
> nodemon index.js

[nodemon] 3.0.1
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,cjs,json
[nodemon] starting `node index.js`
simple-scraper running on 8080

While not a lot of useful features yet, it allows us to have a server that is in fact running and listening on port 8080. Back to coding.

Basic Functionality

Let’s create our first express route handler, which means to say, lets use Express to listen for an incoming HTTP request. We will look for the HTTP verb GET, on the URI "/". In index.js, above your call to app.listen(), add the following function:

app.get("/", (req, res) => {
  res.json("Hello World, from simple-scraper!");
});

Due to nodemon watching for your file to change after it is saved, barring any errors, if you go to http://localhost:8080 on your browser, you will now see that output:

The critical things to note here:

• We are listening on the homepage URI, which is just the forward slash. Future API endpoints will be something else as you will later.

• As we are building an API, we return the data as JSON via .json().

Grabbing a Web Page and Checking It Out

Now it’s time to add an endpoint to grab a webpage and do something with it. For now, we will just pick the front page of Reddit.

Add a new route, this time named "test-route". In this route handler, we are going to use the axios library to make a call to Reddit to retrieve the contents of the homepage. Your code will look something like this:

app.get("/test-route", (req, res) => {
  axios.get("https://www.reddit.com").then((response) => {
    const html = response.data;

    console.log(html);
  });
});

This is just to show you how to make the call in axios and then get to the HTML in the response object (via a promise-then async pattern, meaning we want to wait for a response from Reddit, then we do something about).

Lets now add cheerio to the mix, and grab some data from the Reddit homepage HTML and display it back to the browser that made the call to /test-route. We will use the $ select function (as in jQuery). cheerio also recommends as best practice to prefix your cheerio variables with the dollar sign so it’s clear what they are.

NOTE: Substack renders the two-line version of the dollar sign instead of the one-line variant.

In our case we will:

• Find all the links in the HTML.

• Push reddit.com links to an array.

• Push non-reddit.com links to another array.

• Display the two arrays back as JSON on /test-route in the response.

NOTE: Many links that actually are Reddit.com links are placed in the non-reddit.com array because they did not contain reddit.com in the href.

Here is our updated code as described above:

app.get("/test-route", (req, res) => {
  axios.get("https://www.reddit.com").then((response) => {
    const html = response.data;

    const $ = cheerio.load(html);

    $("a", html).each(function () {
      const text = $(this)
        .text()
        .replace(/(\r\n|\n|\r)/gm, "") // remove line breaks
        .replace(/\s+/g, " ") // collapse multiple spaces to one
        .trim(); // remove leading/trailing spaces.
      const href = $(this).attr("href");

      if (href === "#" || href.match(/[./]reddit.com/i)) {
        redditLinks.push({
          text,
          href,
        });
      } else {
        nonRedditLinks.push({
          text,
          href,
        });
      }
    });

    res.json({
      redditLinks,
      nonRedditLinks,
    });
  });
});

Response

When we get our result in the browser, it is returned with a Content-Type header value of application/json; charset=utf-8. This gives us a wall of JSON like this:

We can get a more helpful view of the data with some additional tooling.

JSONVue

A helpful Chrome browser extension named JSONVue can be used to give a nice, collapsable view of the returned JSON data from our service. Now when we enable the extension, we get a much nicer view when we refresh the tab, like this:

Summary

Sweet, you have built your first web scraper. Now what? Some thoughts on this project:

• We are not saving the data anywhere. A database feature enhancement is a must if you’re looking to do any type of analysis on the data over time.

• Text. We only grabbed URLs, but what about content?

• Abuse. Not many sites are scraping friendly, even though the Internet is searched automatically by bots, crawlers, scrapers, endlessly. Be mindful of the resources you are using of the sites you’re looking at.

• Data Security. PII, GDPR, CCPA and other legal frameworks dictate what data is private and which is not. Be mindful of that.

• "if (href === "#" || href.match(/[./]reddit.com/i)) {", this logic and regex leaves a lot to be desired as noted earlier. An obvious enhancement is fixing this up to be more realistic to reddit.com’s structure.

GitHub

In prep for placing this code in GitHub:

• A .gitignore file was created, with a single entry "node_modules". This should be standard on most Node.js projects to avoid copying thousands of unnecessary files.

• A small README.md was added.

The code for this post can be found on GitHub here.

Prerequisites

As we will be using Homebrew (brew) to install Docker Desktop, if you do not already have it, please read how to install and test Homebrew in the first third of this previous post:

Docker Desktop

Today there are many tools and methods for handling working with containers. This post is not a critique on any other tools, rather it’s simply how to get up and running fast with one of the most common. How fast can we do this:

• Open a terminal.

• Enter:

brew install --cask docker

• You will be prompted for your password, so enter it and press enter.

• If everything was ok, you will see:

🍺  docker was successfully installed!

As its always a good idea to test things are good, enter:

docker --version

To confirm docker is installed, reachable and working.

Now to confirm Docker Desktop is working, lets run Docker Desktop via the Spotlight feature:

Presuming it launched, as of the time of this posts creation you will see:

Click "Accept", if you agree to the ToS and then click "Finish" to use the default configuration.

You will be prompted to give Docker Desktop access to finish its configuration. It will also add some items in your "Login Items" configuration in the "Allow in the Background" section as seen here:

At this point you will be prompted to log into docker.com. If you have not already created a docker account, now is the time to do so, so you can interact with hub.docker.com. It is not required to have an account to finish the tasks here, but odds are you will spend some time on docker.com in the future.

MongoDB

Installing MongoDB

First thing we want to do is obtain the MongoDB image to run in Docker. From inside Docker Desktop:

• In the Search bar, type mongodb.

• Click "Pull", to download the latest image. (Tag is set to "latest" by default.)

Once completed it will have a checkmark next to the "latest" tag.

Configure Docker and Run MongoDB

While we have our image, it’s not quite ready. We need to expose the running MongoDB daemon inside the running container to your laptop, so you can access it externally from your code and other tools.

To do so:

• Click the "images" icon on the left side.

• In the mongo Images row, under Actions column, click the arrow icon.

• Expand the Optional settings section.

• Click on "Host port" and enter 27017. We are essentially having our running container mirror the default mongo port.

• Click "Run".

At this juncture, MongoDB is up and running inside your container. To confirm things are working (as the default Logs tab might be too confusing if your new):

• Click Terminal.

• Type "mongosh". This tool, which stands for Mongo Shell, is available inside the container as its part of the Docker image. It will allow us to interact with MongoDB.

• Type "show databases" to see the default set of databases that in your running database:

To confirm your MongoDB instance is reachable from your desktop, in a browser, attempt to reach it by entering https://localhost:27017. The response will come from MongoDB itself.

mongosh

As we currently have no way to interact with the database outside of the terminal, let’s install mongosh on our Mac and use it to interact with the mongo container.

First install via brew:

brew install mongosh

Next, confirm it was successfully installed and works:

mongosh --version

Finally, confirm it can reach our running instance by simply typing:

mongosh

If everything is working fine, you will land on the default "test" database which has no collections by default. The output will look like this:

Now you can utilize MongoDB as if it was installed and running locally. If your mongo container is not running you will get the following error:

NOTE: At this juncture, there is nothing configured in terms of security, so your database is wide open.

Bonus - Docker Without Docker Desktop

Without even knowing it, you have indirectly installed the tools to work with Docker outside of Docker Desktop. For example, there is a hello-world Docker image that will print out some basic information, more as a proof things are work. From your terminal, simply run the command "docker run hello-world" and it will automatically pull down the latest image for you and run it, like this:

rob@Robs-MacBook-Pro-M2-MAX ~ % docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
70f5ac315c5a: Pull complete 
Digest: sha256:dcba6daec718f547568c562956fa47e1b03673dd010fe6ee58ca806767031d1c
Status: Downloaded newer image for hello-world:latest
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (arm64v8)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/
For more examples and ideas, visit:
 https://docs.docker.com/get-started/

It even shows you how to launch a full Ubuntu image and start in the bash shell:

rob@Robs-MacBook-Pro-M2-MAX ~ % docker run -it ubuntu bash
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
db76c1f8aa17: Pull complete 
Digest: sha256:ec050c32e4a6085b423d36ecd025c0d3ff00c38ab93a3d71a460ff1c44fa6d77
Status: Downloaded newer image for ubuntu:latest
root@9a99929621b3:/# uname -a
Linux 9a99929621b3 5.15.49-linuxkit-pr #1 SMP PREEMPT Thu May 25 07:27:39 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

You can view these images and running containers in Docker Desktop too!

Summary

The end result of running a container could have been done in many different ways, including Docker without using Docker Desktop. This tool just gives us a quick and easy way to get going. With the additional step of installing mongosh on your computer, you can now fully interact with your MongoDB instance.

In the previous post on adding versioning and releases to BookRemarks ended with talk on adding additional features (and new releases) in the future. In today’s post, we will:

• Modify the codebase to support IndexedDB to maintain our research sessions across browser restarts and over time.

• Enhance the UI to work against this new data store and its API.

• Cut a new version of BookRemarks in GitHub.

IndexedDB

IndexedDB is a JavaScript API supported on multiple browsers that allows manipulation of a local NoSQL database. To simplify, it’s a way for us to store our data in persistent JSON objects. There are many ways to store data, but any serious software project will need to move beyond things like property (key/value) stores, cookies, in-memory only (does your process never restart?), etc. The IndexedDB API will give us this ability in our Chrome extension.

How to Upgrade BookRemarks

In the earlier iteration, we merely were saving our captured data to an array in sidepanel.js:

sessionData = new Array();

There are many ways one could go about refactoring here to add IndexedDB support. For now, let’s worry less about updating any existing code and add initial support for IndexedDB.

New Branch

First step is to create a new branch for this feature development. We are going to branch off of main, NOT the new tagged release we made in the previous version. We will name the new branch: IndexedDB-Support-Added.

Creating a New IndexedDB Database

Later we will discuss refactoring the code in relation to removing our previous localStorage code and adding code work with IndexedDB. For now it will help to understand some fundamentals about using IndexedDB.

As IndexedDB is an API, there are fundamental "nice to haves" missing. For example, if you want to search, you need to open a cursor and loop through your data and confirm if its a record you want (you can see this in the getActiveSessionFunction here).

To create a reference to IndexedDB and what will become our database, we use the following code (this is for example):

let db
const db_request = indexedDB.open('BookRemarks', 1);

It’s critical to note:

• If "BookRemarks", the database, does not exist, this will create it for us.

• "1" is our version of the database. This value MUST be a positive integer, do not version your code like a float or decimal. Changing your version here would indicate you have updated your schema fundamentally.

• open() will create our database if it does not already exist.

The call to open is to a factory method, which will return the reference to the database to us asynchronously. This means we need to capture the various events that can arise.

Event Handlers

For now we will need to capture onerror and onsuccess from the earlier request. We will output to the console some informational messages how it went:

db_request.onsuccess = (event) => {
    console.log(event.target.result)
}

db_request.onerror = (event) => {
    console.log(event.target.result)
}

When you load up your service worker DevTools window, in the console upon refresh of whatever page you loaded after opening BookRemarks, you will see some output like this (from db_request.onsuccess):

Refactoring

For cleanliness, let’s do a bit of cleaning up. The general flow we are looking for given the files involved is like this (using file names from the project):

blank.js     <---> background.js <---> indexeddb.js
sidepanel.js <---> background.js <---> indexeddb.js
content.js   <---> background.js <---> indexeddb.js

The intent here is direct interactions with IndexedDB will occur in indexeddb.js. Given a Chrome extension does all its inter-script communication via its service worker(s), that critical messaging logic will go in background.js. The UI interactions and user originating events will be of course in the content scripts, blank.js, content.js and sidepanel.js.

This means relative to v0.1.0, we want to:

• Remove all code related to Chrome localStorage, as we will no longer be using it.

• Add code required to interact with IndexedDB.

To see the original code, please view the initial commit here.

We will first need to create the code to initialize an IndexedDB instance. There are third party libraries out there to help make this experience easier, however, I went with the raw API’s to show what is going on under the hood.

• Create ./scripts/indexeddb.js and add the following code:

// IndexedDB
let db;

// db and object store properties
const dbName = "BookRemarks";
const objectStoreName = "sessions";
const sessionKeyPath = "id";

async function initDb() {
  return new Promise((resolve, reject) => {
    let request = indexedDB.open(dbName, 1);

    request.onupgradeneeded = (event) => {
      console.log("idb onupgradeneeded");

      // onsuccess will fire are onupgradeneeded completes
      // so no need to resolve with db here.
      let db = event.target.result;

      let objectStore = db.createObjectStore(objectStoreName, {
        keyPath: sessionKeyPath,
        autoIncrement: true,
      });

      objectStore.createIndex("currentSessionIndex", "currentSession");
    };

    request.onsuccess = (event) => {
      resolve(event.target.result);
    };

    request.onerror = (event) => {
      alert("Error Event, check console");
      console.error(event);
    };
  });
}

// initialize database and session
(async () => {
  console.log("Initializing DB.");
  db = await initDb();
  console.log("DB initialized.");

  // stand up BookRemarks
  getActiveSession();
})();

This code creates an instance of the database when this file is loaded. This is done via the initialization code at the end. Currently you can’t use "await" on the top level so we have wrapped the initialization code in an anonymous function. Additionally we obtain the current session from the database, if it already exists.

• At the top of background.js add the following importScripts code:

try {
    importScripts('/scripts/indexeddb.js');

    console.log("Scripts imported.")
} catch (e) {
    console.log("Failed to load indexeddb script.")
    console.error(e);
}

This will synchronously import the indexeddb.js script when background.js loads, helping ensure the availability of the database to the extension.

Data (Non-) Schema

This is a time to mention that IndexedDB is not a database per se, rather an API on top of a NoSQL database. This means we still have some configuration to do. Earlier we created the database instance for the extension, but what we did not create was the actual data store inside the database to hold our data. This is called the Object Store. Each record (aka a document) represents one object in the Object Store.

Given the NoSQL nature of this type of database, many people presume it means no-schema. This is not true. While the schema can be highly divergent between records in the Object Store, it does not mean its absence of all structure (schema).

In the BookRemark extension, we capture two types of data:

• Bookmarks - Captured during bookmark creation in the browser.

• Content - Selected text and metadata around it.

We have been saving this data in the sessionData array via two captured events, "bookmarkCreated" and "contentCapture". Let’s break down the current structure of the messages (data) sent with these events.

Bookmark Data

• type (bookmark or content)

• url

Content Data

• type (bookmark or content)

• selectedText

• url

• fullBodyText

• date

One critical item missing here in our MVP (minimal viable product) is data about the session itself. No where did we really capture this. We will expand on this concept to group all these items together into a cohesive structure.

Session Data

• id (will use as unique key)

• currentSession

• name (name of the session)

• startDate

• bookmarks[]

• contentCapture[]

This structure captures basic data about the research session as well as groups all the data we collected. A session data record on its own should clearly resemble the over-time data capture activity of the extension.

To get our data structured how we like, we will need to make a few updates. Conceptually we need to now handle:

• Multiple sessions, with only one active at a time.

• Understand which is the current session (needed for loading the correct data).

• Ability to switch between sessions.

• Ability to save sessions.

• Ability to delete sessions.

Fundamentally we are building out the CRUD processes (create, read, update and delete) which will take a combination of data structure and UI enhancements.

Sessions

New Session

When the extension loads, we need to see if a session exists, and if so, is it the active session. If it does not exist, we need to set things up for the new current session.

In indexeddb.js we will create a function name getActiveSession() that is called when the extension is loaded and database initialized.

Alternatively, when you save and close or delete a session, the side panel will refresh accordingly and place the side panel back into a "new" session state.

Changing Sessions

A dropdown will be added, to which its options will contain the session id as their values and sessionName as the option text. Upon selecting a value from the dropdown, the side panel will refresh to show the selected session.

Deleting a Session

If you delete the current active session, it will be removed from IndexedDB and the side panel again will refresh back into a "new" state.

UI Enhancements

The initial version of BookRemarks looked pretty sparse. With the additional functionality added, this was a good time to make the interface better. While third party libraries could really spruce it up, as these posts are about how to build things, I opted to go with pure DOM manipulation behind the scenes in JavaScript (both for the database and the UI interactions).

The plan of attack is:

• Move the core rendering of the side panel timeline out, to its own timelineRenderer.js file.

• Extract related and common functionality to its own functions, like rendering of the bottom toolbar where our buttons are and code related to the dropdown. Also not just UI, but state management, like keep tracking the activeSession and resetting state on 'new'.

• Update the CSS, well, everywhere, to help make it all look nicer.

• Update bookremarks.html, which is our replacement page for the blank tab in Chrome. This page gives us a more spaced out view of our saved session(s) as well as its own functionality, such as allowing us to see the full body of text captured when you select text. This allows you to go back and see what the full text was at the time of capture.

Debugging

Service worker DevTools does not have access to Application→IndexedDB to see the data. You must:

• Open a visual component’s DevTools, inspect your popup, side panel or overridden default page (history, bookmarks or about:newtab).

• If there is no page in your extension, get the ID of your extension from chrome://extensions, and then visit the manifest page of the extension in chrome via:
  chrome-extension://{extension_id_here}/manifest.json
  Then inspect that page to see your extensions instance of IndexedDB.

NOTE: While you can view and interact in the pages DevTools view, you can only directly run code against the instance in your Service Worker DevTools. Meaning if you put a function like "saveDummyObject()" to add test data and want to run it from the DevTools console, it has to be done in the background worker DevTools to have context.

Summary

While on the surface, outside of some UI cleanup and obvious functional additions, not much changed. However, under the hood, the code took a big step towards something more real.

However, there are many aspects of the extension that need looking into, for example:

• As it works on all open Chrome browsers (non-incognitio, but that can be added), highlighted text on one page will be recaptured if not deselected when selecting on another page. Probably not ideal but of course can be handled. Maybe a good time to look into activeTab permissions instead of universal url access?

• Third party libraries. There was no real need outside learning to use the raw API. One could use the popular Dexie library that wraps IndexedDB for us. Same with our DOM interactions, why not use a framework or library designed for it?

• Missing features. Well, we did not exactly lay any out, so are they truly missing? But to be a proper extension, it should do what it’s described to do, and only that. Has research been made fundamentally better yet? Maybe not, but it’s a start.

A root though to these three points is product design. If you do some research on capabilities, along with your own experience, lots of missteps and even needs for refactors could have been avoided with some groundwork done first.

The code for this post can be found on GitHub here.

Versioning and Releases

Previously we added the BookRemarks source to GitHub for source control. Aside from a best-practice (source control) it allows us to start considering what is a release. For now what I will show is less about perfect project management and more just to expose how to do it so you can start to think about it in your case. There are many ways to version (and tag) your release, for example having monthly releases, quarterly releases, releases based on crazy nicknames.

Semantic Versioning

To be more methodical, we will use Semantic Versioning 2.0.0, which is summarized as:

Given a version number MAJOR.MINOR.PATCH, increment the:

1. MAJOR version when you make incompatible API changes

2. MINOR version when you add functionality in a backward compatible manner

3. PATCH version when you make backward compatible bug fixes

Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

Given the above concepts, means we will now convert the earlier code deployed to GitHub as 0.1.0 (0 as initial version overall, 1 for our basic initial functionality, 0 as it’s not fixing anything, though no doubt bugs are hiding somewhere in there). So how to do this with GitHub?

GitHub Release

For now we are going to use the GitHub website to set up our release, though there are other ways to do it (GitHub Actions for example). It’s not a very heavy process however. The process is as follows:

• Navigate to your repo, for myself, this means go over to BookRemarks repo at https://github.com/robpetty/bookremarks.

In the right side of the page you will see a Release section, with a "Create a new release" link:

• On your {repo}/releases/new page, you will want to create a tag. As suggested on the right side, it’s very common to use your version number with the letter 'v' prefixing the string, like v0.1.0. So click "Choose a tag" and "+ Create new tag" and enter v0.1.0 and press enter:

• From here, a few extra things to note:

  • We will use the main branch for now, so leave Target: main.

  • Enter a "Release title" and a "description" you find appropriate.

  • Check the "Set as a pre-release" checkbox. As this is a learning project and not really stable, we want to indicate this is a work in progress. This could also be done in the tag itself, but GitHub will handle this for us by checking this box.

Your screen should look like this:

So far so good, now just click "Publish Release". The result of this operation will be your first release, v0.1.0, and the assets related to your release zipped and tar+gzipped as seen below:

Now when you click "Releases" from the homepage of your repo you will get a view of your releases, by release version, or by tags:

Next Release?

Ok, so now that we did this, why? The original extension is an alpha proof of concept. Now it’s time to start iterating on specific functionality and building a more formalized release process. Now we know how to "cut" a release, it’s now time to start building.

React is one of the most popular and powerful JavaScript user interface libraries available today. The React ecosystem now includes all kinds of helpful frameworks, tools, libraries, etc. The goal of this post is to show how fast one can get started with React and drop knowledge along the way to help you down the road.

Getting Started

Prerequisites

We will be using Visual Studio Code and Node.js as our underlying toolchain. If you do not have these tools installed, if you like to use Homebrew to make installation easy, see this post how to install Homebrew and Visual Studio Code:

As we will use Node.js and not Python, you can ignore the Python details there, and run the following command to install Node.js after installing Homebrew:

brew install node

Why do we need node? Node.js is an open-source server that can run JavaScript outside of the browser. More directly in our case, it includes a module called "create-react-app" that allows us to generate our initial React app nearly instantly. We will also be using the npm package manager from Node.js, as well as npx (node package execute).

Build the App

Now that we have the basic toolchain ready, we will speed through the initial creation and then talk about what happened.

0 - In Visual Studio Code, open a terminal and navigate to your source code folder

NOTE: We will work out of the terminal in VS Code going forward.

1 - Install the create-react-app module

npm install -g create-react-app

2 - Run create-react-app, pass in the name of your React application

npx create-react-app react-test-app

3 - Navigate into your applications folder

cd react-test-app

4 - Start your application

npm start

NOTE: You might be prompted to allow VS Code to access Google Chrome (or your default browser) to perform access, click "Ok"

That’s it! In your browser, on http://localhost:3000 you will see the default index.js page.

In the VS Code terminal you will see the output from npm start, including how to view your page locally as well as on your local network as seen below:

But what actually has happened here?

npx

We initially installed NPX (node package execute, the node package runner), to allow us to execute the create-react-app module we were using in the upcoming steps.

create-react-app

The create-react-app is a Node.js module that quickly builds out a single-page React application as well as setting up the development environment needed to work on your React app. It includes a lot of other features however:

• ECMAScript version support

• Asset handling

• CSS handling

• ESLint analyzing

• Instant reload

• … and other features

npm start

Thanks to the create-react-app module, our configuration was ready to go for launching our React application through node via the preconfigured package.json files "scripts" config. In this case, the "start" script runs:

"react-scripts start"

Launched the node process and automatically handled VS Code opening Chrome tab to "http://localhost/3000".

Application Structure

Let’s talk about the files and folders that were generated for us and what they do.

• node_modules (folder) - npm-installed packages (Do not check into GitHub!).

• public (folder)

  • favicon.ico - icon generally used by browsers to indicate your web page in the tab(s)

  • index.html - React injects your app code in the <div id="root"></div> element.

  • logo192.png and logo512.png

  • manifest.json - contains metadata about your application

  • robots.txt - define what user agents and web crawlers are allowed, or not, to analyze your web site

• src (folder)

  • App.css - for styling your React components

  • App.js - contains the main logic of your React application, the main component

  • App.test.js - contains your unit tests, run by Jest (which was configured via create-react-app, optional)

  • index.css - for global styling

  • index.js - for any bootstrapping needed

  • logo.svg - scalable logo used in App.js

  • reportWebVitals.js - reports performance of your app (optional)

  • setupTests.js - test configuration (jest by default, optional)

• .gitignore- to prevent folders and files from being checked into Git/GitHub.

• package-lock.json- npm-generated file to keep track of specifically installed dependencies (and sub dependencies).

• package.json - node project config and metadata. npm also uses this file to know what dependencies to install.

• README.md - file used to provide project details in GitHub.

NOTES

• Webpack does not read the public folder, so do not place code files in it.

• %PULIC_URL% URI segments tells React to compile into the output (avoid if possible).

Debugging React in VS Code

It’s great we can get this running so fast, but a project of any complexity will require being able to debug it. How can we set up debugging in VS Code?

• Click Run and Debug in the left panel.

• Click "create a launch.json file".

  

• Click Web App (Chrome), this will create launch.json and add a configuration to launch the Chrome browser on http://localhost:8080 as seen below. These steps will add .vscode/launch.json to your project.

• Edit your launch.json port from 8080 to 3000 since we are running on the default React port.

From here, it's just a matter of using it. So if you set a breakpoint on a line in index.js:

• Launch your application via "npm start".

• In Run and Debug, click the green arrow next to “Launch Chrome against localhost" or select "Run Script: start" and press the green arrow.

This instructs VS Code to launch the debugger against the running application on http://localhost:3000. Below you can now see the debugger has hit the breakpoint and waiting on our next commands.

Next Steps

From here, it's up to you what you like to do with your React application and components.

The code for this post can be found on GitHub here.

What is a VS Code Extension?

Visual Studio Code at its heart, is a source-code editor that can do most of the things one expects when working in a given programming language. However, its abilities are not limited to just editing code. It also allows you to extend the IDE in a great many ways, to help assist you in whatever tasks you might have. Many extensions in the Visual Studio Marketplace of course are focused on enhancing your code editing experience. Other extensions are complete tools by themselves, like Postman.

Yeoman and Additional Basic Tooling

While one could build an extension by hand, even Microsoft has a tutorial that utilizes Yeoman, a tool that builds the scaffolding of many different projects. In our case, we will use it to scaffold the basic framework of our extension for us.

Prerequisites

Assumptions here you have installed:

• Visual Studio Code

• Homebrew.

Tooling Installation

You may already have node installed, but just in case we will install here, via Homebrew.

Install Node (node)

brew install node

We need node for a variety of reasons, but initially to get NPM (the node package manager) to install Yeoman.

Install Yeoman (yo)

As Yeoman is a node package, we will use NPM to install yo and the VS Code Extension Generator:

npm install -g yo generator-code

Building Your First VS Code Extension

Now that the underlying frameworks are in place, time to use Yeoman to build our first extension. There will need to be some decisions made as we go along.

First, navigate to the location you want your extension to be placed. Or tooling will create the top level folder for you, so if you have a folder that you place your code in, be sure to run the command inside that folder.

Now run yeoman pointing to the VS Code Extension generator:

yo code

In your terminal you will be presented with an interface asking to fill in the blanks. It will look something like this:

For simplicity we are going to make the following choices:

• Select "New Extension (JavaScript)". Press enter.

  • TypeScript is fine, however at this moment we did not talk about installing the typescript compiler. Also TS compiles down to JS.

• It will now prompt you for a name. I am using simple-vscode-extension for now. Press enter.

• What is your extension identifier? Leaving blank, press enter (will default to your name in question previous to this).

• What is the description, leave blank, press enter.

• Enable JavaScript type checking in 'jsconfig.json'? Type y, press enter.

  • This is nice to have. While we chose not to use typescript for now, we can get additional tooling help from VS Code by selecting yes. Read more here.

• Initialize a git repository? Type y, press enter.

• Which package manager to use? We will stay with npm, so ensure it’s selected and press enter.

At this point Yeoman will work its magic and create several folders and files for you which you can see in the output:

However we are not quite finished:

• Open with 'code' or Skip? Select Open and press enter.

You will now get a new instance of the VS Code IDE opened to your new extension project. You might be prompted (depending on your setup) to install various things based on file extensions of the newly created files in your extension.

In my case, it was recommended to install 'ESLint' from Microsoft, to which I will choose to install now (but is optional for you). (I intend to work out of VS Code primarily, alternatively you could say no, and then run:

npm install eslint --global

To make it available everywhere. ESLint statically analyzes your JavaScript code to help find any errors.

What Was Created?

As you might now expect, a basic VS Code extension, named simple-vscode-extension has been created. Inside the folder is a variety of folders and files so let’s quickly go through them:

• .vscode (folder)

  • extensions.json - tells VS Code what extensions to recommend for installation when working on this project.

  • launch.json - configuration for the VS Code debugger.

• node_modules (folder) - npm-installed packages (Do not check into GitHub!).

• test (folder)

  • suite (folder)

    • extension.test.js - where your tests reside.

    • index.js - configures the test suite (uses Mocha).

  • runTest.js - Does setup to run tests.

• .eslintrc.json - for configuring ESLint.

• .gitignore - to prevent folders and files from being checked into Git/GitHub.

• CHANGELOG.md - for recording any notable changes to your project.

• extension.js - primary extension code goes here.

• jsconfig.json - JavaScript configuration.

• package-lock.json - npm-generated file to keep track of specifically installed dependencies (and sub dependencies).

• package.json - node project config and metadata. npm also uses this file to know what dependencies to install.

• README.md - file used to provide project details in GitHub.

• vsc-extension-quickstart.md - a guide provided describing key files that were generated, how to run your extension, etc. Some details here might not apply (as the files that have been generated have changed over time.

That might seem like a lot but it all boils down to files that:

• Run the code your extension is designed to do.

• Hook into VS Code, meaning the IDE.

• Hook into other tooling, like npm, node, GitHub, etc.

• Describe what we got going on.

Running the Extension

From here, as we only have a source code view of the extension, we need to launch a new VS Code instance with your extension loaded. Simply press F5. What happened?

• A new instance of VS Code is launched, with your extension enabled. You can confirm this in the title bar of the IDE:

  

• The original instance of VS Code is now in debug mode:

  

From here you can execute the default features of the extension given by the generator for us.

Default Features

Our auto generated code has given us one command to run, via the VS Code command palette (⇧⌘P), "Hello World". This command (as in the words to type) is configured in the package.json of your extension under "contributes.commands["title"]". A sibling key here is "command". This key’s value is how you find what the command does. So in our case with the value "simple-vscode-extension.helloWorld" we will be registered in extension.js in the activate method.

The anonymous function tied to this command, is as follows:

// Display a message box to the user vscode.window.showInformationMessage('Hello World from simple-vscode-extension!');

After running "Hello World" in the command palette such as:

You will see an informational dialog open in the bottom right with "Hello World from simple-vscode-extension!" being displayed such as:

Back in your original VS Code instance, in the DEBUG CONSOLE tab, you will also see output from the active(content) function:

Whats Next?

There you have it, the standard out of the box VS Code extension and details about what is going on. As a typical "Hello World" type of application, this is merely the beginning. In a future post we will discuss enhancing the extension to do more realistic tasks, expose additional functionality, testing your extension, how to debug it, etc.

The code for this post can be found on GitHub here.

This post is intended to help understand how to access the Chrome DevTools debugger from various aspects of your Chrome extension as it’s not always straight-forward.

It’s a an on-going culmination of learnings from my own Chrome extension development which you can read about in these earlier posts:

Launching DevTools by Extension Component

content.js (Content Scripts)

By default extension content scripts are excluded from the debugger so they are a bit difficult to find. I was not aware of this initially, so to find my content.js in an earlier project I used a trick to discover it. I placed a console.log() message at the top of your script, so when I opened the DevTools on the web page I was debugging, that message got me the link to the source file I needed.

This gives you a hook, by clicking content.js, that will now open the Sources panel view of this file and prompt you to either remove from the ignore list or give you an option to Configure the list.

Here is a view of the script waiting on your decision:

If you click Configure it goes to the DevTools Settings tab (which is the little gear icon on the right in the toolbar:

You also don’t have to make a choice here. You can now debug the file without changing your environment (for example you can now add a breakpoint and refresh the page and it will be hit).

background.js (Background Worker)

In chrome://extensions, click “Inspect views: service worker” to get debugger for just this script.

sidepanel.js (Side Panel)

Right-click once open, click inspect.

NOTE: If you have other DevTools running, like one for the background worker, you will be able to see it in this view as well.

popup.js (Popup)

To debug the popup page and its scripts, simply right-click on the extension icon and select "Inspect Popup".

Page Override (Bookmarks, History, about:blank)

These are just regular pages, once your new “blank tab” page is loaded (or history or bookmark replace pages), right-click on the page and go to inspect. From here you can see your resources and scripts as normal.

Summary

Hopefully this page has saved you some time finding this critical development tool. Down the road I will post about how to go about using the debugger to help speed up solving your code issues.

If you are like lots of people I know, you spend too much time organizing bookmarks in Chrome with the hopes that it will speed up finding some bit of data sometime in the future. You also might be doing research and want to keep some sources organized based on some important text you found.

Is there a better way, or at least a way with less manual effort, one can do? Let’s find out!

In this post we are going to build a basic bookmarking research extension that saves aspects of the data you care about. I am calling the extension BookRemarks!

Problems We Are Solving For?

The number one issue I face with bookmarks is the cumbersome nature of managing large volumes of them, along with remembering where some key detail was I cared about on their respective pages. With those two issues in mind (and whatever else comes along), some design goals here in relation to these are:

• Keeping the extension itself from being burdensome. When you capture some text you like, ideally we will:

  • Capture the full textual content of your selection.

  • The URL of the page you are on.

  • When you captured the copy. This will help in our organization. (Not perfect of course as you might look up multiple topics at the same time.)

• Ease of access. The less steps to get back to your data, the better.

• Enriching. How can we do better than “somewhere we have a bookmark in hopefully the correct folder”?

What we are not doing:

• Replace core Chrome bookmarking. They are safe, for now.

Basic Architecture

No surprises here if you have see my previous posts on building Chrome extensions here:

and here:

Building off what we have learned, initially this extension will work as a side panel extension, with additional functionality, including saving details locally.

Basic Architecture

Here will be our initial file layout:

• manifest.json file

• popup.html

• content.js

• README.md

• scripts/background.js

• sidepanel.html

• sidepanel.js

Like I said, no surprises here. So what will be new?

• bookremarks.html, we are going to override the default blank new tab with our customer bookRemark.html. Intent here is you can get a view of your research soon as you open up the browser and a new (blank) tab.

• blank.js, the script file we will use in bookremarks.html (named in our of the about:blank tab we are replacing later).

• styles.css, to spruce the place up.

We will also be:

• Accessing additional APIs, more as the post progresses.

• Accessing a data store. Got to put this stuff somewhere!

Replacing "about:blank"

Chrome gives you options to replace various functionality including replacing the bookmarks page, or history and in our case the "new" blank tab, functionally referring to "about:blank". You will still be about to go to a true blank tab going directly to the blank endpoint. However, using this technique we will be able to load a custom HTML page when you fire off a new tab. For now, we will put a placeholder that looks like this:

Why, though? The intent here is so when you open a tab you will be able to see a more complete view of the work you have been doing, versus what will be captured and shown in the side panel view.

How was this done? Aside from creating your custom HTML page (I called this bookremarks.html), you need to update your manifest to include a reference to your page and which functional section you are looking to replace, in our case, the blank tab:

"chrome_url_overrides": {"newtab": "bookremarks.html"}

More info on overriding pages in Chrome can be found here.

Functionality

It’s not enough to capture text we are interested in, we are looking to capture metadata about the research session, the URLs involved, etc. While we did note we did not want to replace basic bookmark functionality, we are interested in capturing them. So how can we do that as they are created?

Bookmarks

The chrome.bookmarks API requires:

• The "bookmarks" permission in the manifest.

• Code related to working with them will be placed in the background worker (background.js in our case).

For now, lets just print out what was captured to get a sense of how one access the bookmark data:

chrome.bookmarks.onCreated.addListener((id) => {
    console.log("chrome.bookmarks.onCreated")
    console.log(id)

    chrome.bookmarks.get(id, (result) => {
        console.log(JSON.stringify(result))
    })
});

With this code, once a bookmark is successfully created, we can capture its id (which may also be a bookmarks folder id as well). Using the get method, we can pull up details about the bookmark. We will later add this data to our research session.

Capturing Text, Metadata and Saving It

The primary function of this extension was intended to:

• Capture selected text, as it’s of the most interest.

• Capture metadata about the text, such as the page URL, when you copied it, etc.

• Save this detail to a data store.

So let’s break this down:

content.js - Data Extraction

To interact with the DOM of a page, we need to place our code in the content script. Additionally, for the code to be injected, we need to set permissions to allow this:

"permissions": ["bookmarks","tabs"],

bookmarks were added previously. tabs allows us to get the url and other details.

We will capture the details we are interested in and send it on via sendMessage for our side panel to process. Here is what the code looks like to capture the selected text, the full text of the page itself, the url of the page, and the time we captured this detail, all before sending onto sidepanel.js to visualize:

document.addEventListener("mouseup", function (event) {
    const target = event.target;
    const selection = window.getSelection();
    const selectionAsString = selection.toString().trim()
    const fullBodyText = document.body.innerText

    if (selectionAsString.length > 0) {
        console.log(selectionAsString)

        data = {
            'selectedText':selectionAsString,
            'url':document.location.href,
            'date':new Date(),
            'fullBodyText':fullBodyText
        }

        chrome.runtime.sendMessage({ type: 'contentCapture', data: data });
    }
});

sidepanel.js - Session Visualization

Before anything, the side panel requires the “sidePanel” permission to be added to your manifest permissions array.

Once a message is received from the content script, we need to take the data and place it as we would like in the sidepanel.html. The design choice here is to create a timeline type view that will take the selectedText out of the payload and place it into the timeline on the sidepanel.html page. Note, there are many ways to do this and I chose direct DOM manipulation here for streamlined control.

Aside from visualizing the data, this is where we will retain our session until time to save it. Again, this data could reside elsewhere, but for simplicity, we will use a new array named "sessionData".

The flow of the sidepanel.js script is:

• On script load, register a function, onDOMContentLoaded(). We will use this function to set the header of our session, in this case the time the extension launched. This function also registers our "Save Session" button and saveSession() function that we append to the "bottom" div of the sidepanel.html page.

• onMessage listener, where we wait for onCompleted events either from bookmark creation, or selected content capture. When either are hit, we will render aspects of the content to the timeline. This function also appends the data to our sessionData array defined earlier.

• createSeparator(), addNewDivWithSeparator(), helper functions for rendering our content.

• saveSession(), will send our sessionData out to be captured by the background.js script, as it has access to localStorage, our data store of choice.

Here is an example of the side panel with some text captured:

Additionally some care was taken to prevent duplicates. This might seem obvious but:

• A text selection could be a duplicate of one much further down the page and not obvious.

• Double-clicking when text is highlighted causes a capture, but of the highlighted text, which could be a duplicate and easy to happen clicking around a page.

On to saving the data.

background.js - Data Management

The background worker script, background.js is focused primarily on our localStore database interactions with one exception.

Bookmark Creation Capture

During your working session you might want to bookmark the page. That’s perfectly fine to do while using this extension. However we can capture that operation to send the bookmark details on to the side panel where we can save and visualize it. To do this we add a listener to chrome.bookmarks.onCreated.

localStorage Database Management

The flow of managing our data in background.js is as follows:

• On loading the script we remove the data we placed in storage. As this version of the extension is just for learning it allows us to keep things clean for now. (NOTE: When you remove the extension, it also removes your data. Otherwise, across browser reboots, it will remain.)

• Getting data. We listen for a message with action set to "getSavedText", which comes from the loading of the bookremarks.html page, to retrieve our session data and display it on the page.

• Setting data. We also listen for "savedText", sent from clicking the "Save Session" button on our side panel, to save the data that is passed into localStorage.

“about:blank”/New Tab - Overriding Default Pages

While we could have chosen to build out a more robust popup.html, secondary side panel, edit options on the original side panel, etc, I decided to show how to override one of the basic Chrome components, in this case the "new tab". Earlier I showed how to do a basic replacement of this page, now it’s time to add some functionality to it.

Upon opening a new Chrome tab, instead of the regular blank (or other option you setup) page you would see, you will now get a view of the data collected by the extension. As one of the options is the full text of the page, I wrote a few small helper functions to strip out newlines, remove HTML, and collapse whitespace to one space for ease of viewing. Now after another small session and saving that data, upon opening a new tab, we see:

What’s Next?

That seems like a lot, but once you check the code itself in GitHub, you will see it’s not so bad. Did we solve bookmarks, or the actual issue, finding the data that we bookmarked? Um, a little bit.

So now what? The first bit was to learn how to further build out your own extensions. This post included:

• Overriding basic Chrome functionality.

• Saving data in a more permanent location (but is it the best?)

• Sending data via messages between the various components of your extension.

• Showing that most of what you see in Chrome, are just web pages, so can be interacted with directly via JavaScript.

Seeing these things can now lead to the next level of features:

• Data stores. Instead of saving data in the localStorage database, what about IndexDB (which given we are using the larger full text of pages in our objects maybe would be a better choice), or even sending the data to a service?

• Data considerations. Do we really need to get the full text for each snippet? Sure the page might change in time, but do you expect it to change during your session? Maybe rearchitecting the data format here would be a good area to focus.

• Better visualization, a given!

• Could more be done with popup.html?

• Sharing your bookmarks and research?

• Print the data to a PDF?

• This extension works on all browser instances and all tabs once open. Maybe in some cases this is not ideal?

The code for this post can be found on GitHub here.

Next step, versioning:

Risky Business

Most modern browsers offer a wide array of extensibility options in the form of extensions (sometimes called plugins). Extensions are generally obtained through an extension marketplace such as Chrome Web Store (for Chrome) and Firefox Browser Add-ons (for Firefox).

A major concern however with extensions is trusting them. For most people, going to a web store and clicking “Add to Chrome” and accepting any permissions needed is about a deep interaction outside the extension they will go to. This is reasonable behavior, but risky. This risk is magnified greatly if you install extensions from third party marketplaces as well as one-off extensions, like the one we are making today!

For more information on Chrome extension packaging, look up the CRX format.

Basic Architecture

Today’s post is an iteration of “Guide: How to Build a Simple Chrome Extension”. If you have never built an extension locally before please give it a read to get your footing.

It’s one thing to capture data, it’s another to visualize it. The goal of the extension is to not only capture requests made by Chrome as requests complete, but also clearly display them so you can possibly take some action. What kind of action? Perhaps you have a local firewall and you like to block some particular tracker. This extension will show you the precise domain the request is being made out toward.

The general file layout will remain the same as our previous extension:

• manifest.json file.

• popup.html file.

• content.js script (unused but required).

• README.md (not a Chrome extension requirement, rather for GitHub).

However there are some new files we need for the additional functionality. What are those?

• background.js in the scripts directory. This is where we will capture the data.

• sidepanel.html. How we will visualize our extension, via Chrome Side panel functionality.

• sidepanel.js. A script we will use to receive messages from background.js. We then take those messages and display the data in sidepanel.html.

Let’s talk about what is going on with these files one by one.

manifest.json

This file is the heart of the configuration to your extension. Key (new) items required for our extension:

• permissions. We will need two new permissions:

  • webNavigation: Required for background.js (our background worker) to interact with the chrome.webNavigation API.

  • sidePanel: Required permission to interact with the Chrome Side panel functionality.

• side_panel:

  • default_path child key with a value of sidepanel.html. This is so Chrome knows what HTML file to display in the side panel.

scripts/background.js

background.js is the background worker of our extension. This file will work in tandem with the Chrome APIs available behind the scenes to obtain data, and your visualization components (which in our case is the side panel we are building, but more generically with content.js scripts that interact with the pages rendered).

This file (available in GitHub) essentially flows like:

• Initialize some variables (domainData and urls) used to track where our requests are going.

• Print a message to the console that the background.js is loaded (more on debugging later).

• Register a listener. Once registered, listen for webNavigation requests to complete, process them to get the base domain of the URL, and save that data locally. Send the updated key/value object (domain and how many times it was hit) to sidepanel.js,

What is a Listener?

An extension’s files have restrictions based on permissions, which define what functionality goes where.

A background worker is specifically prevented from accessing the page’s DOM. What it is allowed to do is interact with the various browser APIs behind the scenes. In our case, we want to add a listener to a specific event. A listener is specifically a function registered to listen for one or more events in the lifecycle of an API.

To listen to general network activity, we want to listen for the “onCompleted” event. To add a listener to this event, we add code that looks like this:

chrome.webNavigation.onCompleted.addListener();

Our function will be passed anonymously to it. Essentially we know some data will be passed to the function when “onCompleted” is emitted. We can call this whatever we like, so for now I named this parameter “details”. So now the generalized structure, and adding printing of this data out in a format you can see, would like like:

chrome.webNavigation.onCompleted.addListener((details) =>    {console.log(JSON.stringify(details)});

Debugging background.js

How do we debug the background.js file? Unlike debugging a web page simply by opening the Developer Tools feature, you need to fire off a separate instance for background.js. To do this (presuming you have loaded your extension already):

• Navigate to chrome://extensions.

• In Inspect views click “Service worker”.

  

This will open a dedicated DevTools instance, like this:

background.js Functionality

As noted in the flow earlier, code in this anonymous function is where we will slice and dice the url property from the “details” data sent to our function once onCompleted fired for a given request.

Key items to note here:

• Grab the url and obtain the hostname.

  • NOTE: Some resources will fail, some will be from cache, some will be malware, etc. Meaning the URL might not contain a proper hostname property. In this case we use the entire url. Example is “about:blank”.

• Save the parsed value as domain and add it to the url object.

  • Increment the hit count if we have seen this domain before.

• Order the domains by number of hits each as received so we can see most to least in the side panel.

• Send this ordered data object to the sidepanel.js file, by utilizing chrome.runtime.sendMessage(sortedDomainsByCount).

sidepanel.html

To visualize our data, we need somewhere to place it. As we opted to use the Side panel functionality of Chrome, we simply need to create a basic page. For this extension we are simply making a labeled table, that looks like this:

Aside from visualizing, it’s critical to note, in this HTML is where we refer to sidepanel.js, the JavaScript script we will use to receive information from the background.js worker and update our HTML to view the data.

sidepanel.js

The flow of sidepanel.js is:

• Log the file was loaded.

• Add a listener, this time for the onMessage event.

• A helper function for better visualization.

To add more color to what’s happening here. When we receive a message from the background worker, we want to take the data passed to us, and use it in our anonymous function. We will take that data, re-render the table back to its default state (as with each onComplete we fire out an updated url object.

DOM Accessibility

As noted earlier, different scripts have access to different API functionality based on their permissions. sidepanel.js works much like a content.js script, it has access to the DOM. In this case, not a regular web page’s DOM but instead the DOM of our sidepanel.html.

This allows us to manipulate the HTML directly. In our case we simply add a new row to the table, and in the first column add the domain, and the second add the count.

Visualization Tweak

Some domains (especially due to subdomains) get really long. This will cause the side panel to add a horizontal scroll bar and push the counts off the screen pretty quickly. This is not ideal behavior. A basic remedy we can do is truncate the values.

This is why we added the truncateDomain(domain) method. It is currently designed that:

• If a domain has more than 30 characters, and more than 2 segments (meaning it detected at least two periods in the domain)…

• Grab the subdomain and grab its last five characters…

• Then return the subdomain prefixed with “…”, and reattach to the base domain.

Not a perfect solution, but intended to show how to make a basic visualization a bit better.

Security

This post is by no means complete in terms of security, but there are several security related items to observe here:

• This extension is watching aspects of your network traffic. It’s intended to show how this type of activity is possible.

• Changing the data. While we simply parsed the domains for visibility, it shows how data can be changed, aggregated, etc before being sent.

• The extension is not limited to any one URL, meaning once installed and active, it listens to all instances of Chrome and all tabs within (with exception of incognito tabs, which could be watched as well but require a special property to do so). This is done via a config in the manifest.json:

  "content_scripts"."matches": ["<all_urls>"]

• Data is captured temporarily, locally. Per the source code in GitHub you will see this data is not stored in any local data store (cookies, database, etc), or sent off on the network to a third party service. That is the next step an extension would take to capture your data on a more permanent basis.

Wrap Up

This post is not to scare you from ever using extensions. It was intended to give a sense of what is possible, how to do actionable and realistic things with an extension and give you pause for concern of what could go wrong. Extensions are a great way to make your day to day life easier! Like all things tech related, be mindful of the security implications, for your data, and others!

The code for this post can be found on GitHub here.

Prerequisites

I will presume you have created a small project that you would like to add source control to. If you like you can follow this post to build a small Chrome extension to use (as we will here). Alternatively if you do not have anything ready, all you need to do is create a folder and inside that folder create a README.md file, like this:

Setting Up Authentication

First step here is to confirm you are logged into the GitHub Pull Requests and Issues extension. You can confirm by clicking your accounts icon in the bottom left and confirm if you see a message to log into it, such as this:

Clicking here will redirect you to your browser to log into GitHub and then prompt to open the *.dev link via Visual Studio Code. Accept this and VS Code will refresh and you will be good to go.

Repo Initialization

The first step towards getting your local code into GitHub is initializing a local git repo for your code. Open your desired directory via Visual Studio code if it’s not already open. On the left side, click “Source Control”, you will see a screen like this:

From here, it is as easy as it looks, click “Initialize Repository”. The view will now change, but what happened?

In your terminal (if not already, you can open in VS code via Terminal→New Terminal), you can type ls -a to see a new directory has been created named .git, and in it are several new files and folders. This folder is how git will maintain your version control history, credential configuration, etc.

In the earlier screenshot, you will see a variety of items. Critical things to know here:

• Nothing is actually in source control yet, this is just preparing it.

• Default is placing the files into an initial branch called “main”. You can also see what branch your on in the bottom left corner of VS Code:

  

• The files that we can choose to add now have a letter U, for untracked.

The next steps here are to take our existing files, and get them into the local main branch on our computer. How do we do this:

• First step is to stage the changes. This means we are adding them to the main (our current) branch. This is done because as your code becomes more complex you might not want to include all new/updated files into this branch. In our case we want to add all, so to the left of the blue circle (which shows number of changed files) click the + sign to “stage all changes”:

  

• The tree will now change, to include a “Staged Changes” folder with our files, and the “Changes” folder now empty beneath it:

  

Now it’s time to commit our staged changes to the main branch! As is tradition for many repositories around the world, a simple message of “Initial commit”, or anything you like to describe your initial code/function of your branch, can be added to the Message text box. To commit:

• Add your Message. In my case “Initial commit”.

• Click Commit.

This results in:

Disaster! What happened here?

I left this intentionally just to show how to do some light git debugging and to help understand how to properly configure. So click “Open Git Log” to see what happened.

Debugging Git

If you scroll up a bit you will see the fatal error relating to this message as well as some other messages about things not configured properly. In this case, the primary section concerns our identity:

2023-07-28 13:22:01.958 [info] Author identity unknown

*** Please tell me who you are.

Run

  git config --global user.email "you@example.com"
  git config --global user.name "Your Name"

to set your account's default identity.
Omit --global to set the identity only in this repository.

fatal: no email was given and auto-detection is disabled

Git needs proper details to help track changes in the files it is tasked to version. Luckily in this case, it literally tells us what to do. As this is our local development host and odds are we are working out of our own personal GitHub account, it’s safe to use the —global flag when setting these things up. To resolve this:

• In the terminal run, after updating to your GitHub email address.
  git config --global user.email "you@example.com"

• In the terminal run, after updating the name:
  git config --global user.name "Your Name"

If you were curious and checked your directory for .git/config you will notice no update there. This is because we are using the global setting which will be in your home directory, so you can view like this:

rob@Robs-MacBook-Pro-M2-MAX .git % cat ~/.gitconfig 
[user]
        email = "you@example.com"
        name = "Your name"

Now that we have updated, please click Commit again and watch your initial files land on the "main branch”.

Publishing Your Repo to GitHub

You might have noticed the button changed to “Publish Branch”.

As there doesn’t seem to be much else to do, let’s click it. What, failed again?

Ok not so much a failure, rather as it’s our first go around, we need to allow the extension to work directly with GitHub so click allow. You will be prompted to allow the extension to open the authentication endpoint, so click Open:

You might miss what’s next so be careful. VS Code will prompt you to add either to a private or a public repository. Given I am creating this post, I will place it in a public repository.

Immediately, barring any issue, you will multiple dialog boxes pop up, such as below:

VS Code Dialogs After Publish Branch

The critical one here is the middle on, go ahead and click “Open on GitHub” to see your now hosted repository there:

The URL will be basically your account name/folder name, so in this case:
https://github.com/robpetty/simple-chrome-extension

Notice your README.md, becomes the README file you see on the repo default landing page in the “<> Code” section.

As to the other dialog boxes:

• As we just published our initial code to GitHub, there are no updates to make so we do not need to “Create Pull Request…” at this time. A pull request is a merge request, meaning to pull a new branch (maybe has files for a new feature, or updates to existing code) back into the main branch (or parent branch if you have a large branching codebase). So just “X” out of it for now.

• VS Code is asking to run “git fetch” periodically. This is good if you’re working with others, so you can see code updates that have been merged. You would want to know this if you need to update your working branch to include some new code, or looking to avoid conflicts, etc. So to this, click “Yes”.

Summary

VS Code, GitHub and Git combine to make a powerful feature set for keeping your source code safe and its progress understandable. Additional information can be found here.

There is a great detail more one can do with these tools, but that’s for another day, another post.

The goal of this post is to share how to build a simple Chrome Extension and give clarity to various aspects of how this is done, limitations, expose features, etc.

To see some of the hundreds of thousands of Chrome Extensions, please visit the Chrome Web Store.

Basic Architecture

The basic extension architecture consists of a mix of folders and files. For our example extension, that we will call “simple-chrome-extension”, contains the following file system items, starting with the root folder:

• simple-chrome-extension (folder)

  • manifest.json

  • scripts (folder)

    • content-script.js

NOTE: You can structure your extension in many ways, the only true requirement is manifest.json must be in the root of the extension folder. Additionally, there are many other files not listed here, as extensions will vary.

So what do each of these files mean:

• manifest.json - The confirmation file of the extension. Everything from the name, its current version number, what icons to display, Chrome API keys, permissions that are needed, what file represents the extension service worker, option popup HTML, etc. See manifest file format and manifest examples for more details.

• content-script.js - Contains the code that is injected into the web page. This is where your code will reside if you need to interact with a web page directly via the DOM. A content script does not have full access to the Chrome APIs, but you can interact with the full API via messaging with the service worker.

There is another key file I will call out here even though we will not use it in this example, the service worker script.

• service-worker.js - The core script that will run in the background. The service-worker.js specifically is to organize the functionality of the extension and listen for events. It is isolated from the page itself so is unable to directly interact with it. See Chrome Extensions API.

You can find other relevant files and how they work in the architecture overview docs here.

Extension Buildout

So what are we building? This extension will simply replace a word you select, with the word “Boof!” in honor of the small dog this blog is dedicated to. The first thing you want to do is create a folder named after the extension, so in this case, in my Repos folder:

/Users/rob/Repos/simple-chrome-extension

manifest.json

As noted earlier, the most critical file (and location of said file) is the manifest.json, placed inside the root of the folder we just created. By later selecting this folder Chrome will then look inside it, for the manifest, to properly load your extension (which we will get to later).

Here is the contents for this simple extension’s manifest:

{
    "manifest_version": 3,
    "name": "Simple Chrome Extension",
    "version": "1.0",
    "description": "Changes a clicked word to say Boof!!!",
    "permissions": [
        "activeTab"
    ],
    "action": {
        "default_popup": "popup.html"
    },
    "content_scripts": [
        {
            "matches": [
                "<all_urls>"
            ],
            "js": [
                "content.js"
            ]
        }
    ]
}

A few observations here:

• Note we called the content script file just “content.js”. We can call it anything you like, it’s just a common practice to use content-script.js as its name but is not a hard requirement.

• We are targeting version 3 of the manifest format, though earlier versions are still supported.

• In terms of permissions we only added activeTab so it can give the extension temporary access to the page when loaded. More details can be found here.

• We left content_scripts.matches with a value of “all_urls”, which actually could have been left out due to the activeTab permission. This was left purely for our local development.

• The popup action, which refers to when you click your icon in the extensions toolbar opening a popup, this is the popup it opens.

popup.html

Speaking of the popup, here is the HTML for this file:

<!-- popup.html -->
<!DOCTYPE html>
<html>
  <head>
    <title>Simple Chrome Extension</title>
  </head>
  <body>
    <h3>Simple Chrome Extension</h3>
    <p>Click a letter in a word, and get barked at by Mya Papaya!</p>
  </body>
</html>

content.js

Now for the heavy lifting (not really). Our content.js will be the file that is allowed to interact with the DOM of the page, so this is where we will place our Javascript code to do our interaction. Of course, the interaction is to change the word clicked to say “Boof!!!”. Here is the code along with some helpful comments describing it:

// content.js
document.addEventListener("click", function (event) {
    const target = event.target;
    const clickedChar = getClickedCharacter(event); // Get the clicked character
    const parentElement = event.target.parentElement;
    const textContent = parentElement.textContent;

    // Find the start index of the word
    let startIndex = textContent.lastIndexOf(' ', clickedChar.index) + 1;

    // Find the end index of the word
    let endIndex = textContent.indexOf(' ', clickedChar.index);

    if (endIndex === -1) {
        endIndex = textContent.length;
    }

    const clickedWord = textContent.slice(startIndex, endIndex);

    // Replace the clicked word with the new word
    const newWord = "Boof!!!"; // Replace with the word you want
    const newTextContent = textContent.slice(0, startIndex) + newWord + textContent.slice(endIndex);
    target.textContent = newTextContent;
});

function getClickedCharacter(event) {
    const selection = window.getSelection();
    const range = selection.getRangeAt(0);
    const clickedCharacter = range.startContainer.textContent[range.startOffset];
    const clickedIndex = range.startOffset;

    return {
        character: clickedCharacter,
        index: clickedIndex
    };
};

Deploying the Extension Locally

In this post we are only going to discuss running this extension locally, not sharing on the Chrome Web Store. You can however share the extension with others, as long as they enable development mode in their instance of Chrome to run it.

One way to add your new extension to Chrome, using one of the Chrome URL endpoints, is simply open a new tab and navigate to:

chrome://extensions

Depending on your browser, Chrome will look something like this:

See in the top right corner? You will need to click the “Developer mode” slider to unlock Chrome to allow your local extension to be loaded. The page should have updated to include additional features to load, pack, and update extensions, as well as new functionality on any installed extensions, such as you see here:

From here all you need to do:

• Click Load unpacked.

• Navigate to your extensions folder, in my case, /Users/rob/Repos/simple-chrome-extension.

• Click select (on the folder level, not any file inside it).

Barring any mistakes, you will see it loaded in Chrome like this:

A few things to note here:

• You can remove your extension here by clicking Remove.

• You can disable it by clicking the extensions slider.

• If you update something, like your description in the manifest.json, you can refresh here by clicking the refresh icon (after you save your file).

• “Details” will open a detailed view of your extension where most of the main page functionality is duplicated, but also you can control things like what sites it can access, if it works in Incognito mode, works against file system files that are loaded in Chrome, etc.

Using the Extension

While we are not setting the world on fire here, it’s still good to confirm things are working as intended. First, you can check the popup by clicking your extension icon, the result will look something like this:

To validate the functionality, we are going to go to https://www.lipsum.com/, which currently looks like this:

Given the extension is ready to go, now clicking a few words, you can see some inspirational “Boof!!!” from Mya Papaya:

Wrap Up

Of course this is a trivial example but it was critical to understand the underlying basics of an extension. Next steps here would be of course to expand the extension to be relevant to your needs, including interacting outside of Chrome itself!

The code for this post can be found on GitHub here.

Update

Here is a link to a slightly more advanced Chrome extension, one that can listen to all the network requests and list out their domains in the side panel while you browse:

Prerequisites

A few presumptions will be made here:

• You are on a Mac.

• Homebrew and Visual Studio Code are installed. If not please see:

  NOTE: You can skip installing Python if going by that previous post.

• You have an account on Github.com. If not, join here.

Initial Steps

Git Installation

For any git source control to work, you must first install git itself. Using brew, this is as simple as opening a terminal and running:

brew install git

NOTE: Details can be found on the formulae itself here.

Confirm your installation by running:

rob@Robs-MacBook-Pro-M2-MAX ~ % git --version
git version 2.39.2 (Apple Git-143)

GitHub Pull Requests and Issues Extension Installation

The official Visual Studio Code extension for working with GitHub is the “GitHub Pull Requests and Issues” extension, which can be found in the marketplace here. Installation will be done here directly in Visual Studio Code. The steps are:

• Open Visual Studio Code.

• Click Extensions on the left side of the window.

• Start to type the name of the extension. As the list filters, you will see it finally at the top as to not get confused with the many other git related extensions

• Click install.

Once completed the informational page will look like this:

Extension Configuration

NOTE: For this example I have logged out of GitHub.

Now we need to connect your local git setup to GitHub via the extension. First, on the left side of VS Code, click the GitHub cat icon. We are going to clone a project called “The Algorithms”, which is a popular Python project for learning how various algorithms are made in Python. The GitHub page can be found here.

As this is a new setup your screen will look like this:

Using the VS Code Command Palette (⇧⌘P), we can clone this repo by typing:

Git: Clone

Once you select the command, enter the URL for the repo which is:

https://github.com/TheAlgorithms/Python.git

Now press enter or click “Clone from GitHub”.

You will be prompted as to where you would like to store this repository (repo). In my case I am using:

/Users/rob/Repos

Click to your preferred top level folder then click “Select as Repository Destination”.

You will be prompted to either open the cloned repo, or add to your current workspace. For now just click “Open”. This will refresh Visual Studio Code and reopen, with the Explorer view rooted at the folder “Python”. This folder lives inside /Users/rob/Repos in my case and is called Python because the .git file was named that.

Your Visual Studio Code instance will now look like this:

Summary

There is a lot one can do with Visual Studio Code, GitHub and various extensions. In a future post, we will continue the journey of how to finish initial git configuration and then create a local repository and push it to GitHub via the tooling installed here. We did not make use of your GitHub account yet, that was just preparation for the next steps.

Where is the Screenshot Feature?

Presuming you have launched Chrome, the first thing you will want to do once you are on the page you wish to capture, is launch the Developer Tools. There are multiple ways to do this.

• On a Mac, command-option-i, will open the Developer Tools panel.

• You can also use the three ellipses, aka the kebab menu, in the top right corner. Click the three vertical dot button→More Tools→Developer Tools, as seen below.

  

From this point you have multiple ways to get to the screen capture tool(s):

• Press command-shift-p, and type screenshot.

• Press the kebab menu button and click “Run command”. Then type screenshot as before.

Screenshot Modes

There are currently four modes to work with in the screenshot tool.

Capture Area Screenshot

A more traditional style of capture, where you will be presented with a crosshair to select a visible area of the page for capture.

Capture Full Size Screenshot

To capture the entire page, even the part beneath the fold, you would use this option.

Capture Node Screenshot

To capture a specific DOM element, you would first select it while the debugging tools open. Two options are:

• command-shift-c, then select the item of interest.

• In the debug tool menu strip, click the cursor on the box icon in the top left, then select the item of interest.

In this sample image I clicked the search box on the default page of Chrome.

NOTE: You can also navigate the DOM in the Elements tab to highlight your node of interest before running the node screenshot capture.

Capture Screenshot

The most basic option, to capture the currently displayed viewport portion of the page (which may be incomplete) can be captured by this option. So in this example I moved the bottom of my browser up to cut off part of the page to show the impact versus using the previously mentioned “Capture Full Size Screenshot”.

This feature set has helped save me lots of time when having to bulk produce images of web pages versus externalized tools. To be clear, there is still need for other tools (for example if you wanted to open the Developer Tools and take an area screenshot of the tools themselves).

Homebrew

First up, you will need a package manager. Homebrew has become the de facto choice recently on the Mac platform. First, you need to open a command line somewhere, either:

• Open Launchpad, “Search” for “Terminal”, click it, or…

• Open Spotlight search field, by pressing command + spacebar and type “Terminal”, press enter (or click it in the dropdown).

Now it’s time to actually install Homebrew. On the command line, paste the following command:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

This command tells the bash shell to take the following command string in quotes and execute it, in this case, telling curl to go fetch the install script and run it. The reason for getting Homebrew is that it will make dealing with Python and other tools easier down the road.

You will be prompted for your password to allow sudo access for installation. It will ask you to install the Xcode Command Line Tools, to which you will press ENTER to agree and continue. These are tools to help developers do things on the command line.

At the end, you will be given more details for next steps:

The critical item for now is ensuring your environment is set up correctly. The default shell on modern Macs is zsh, so its advised to run the following as suggested (note, you will need to change the path to your local path, not mine):

(echo; echo 'eval "$(/opt/homebrew/bin/brew shellenv)"') >> /Users/rob/.zprofile

eval "$(/opt/homebrew/bin/brew shellenv)"

These commands will set up your zsh shell to work with Homebrew by adding it to your path. You can confirm this worked by running the following and seeing that homebrew tooling was added to your PATH shell variable:

env | grep PATH

To confirm Homebrew now works, check its version via:

brew -v

NOTE: The command name is brew, NOT homebrew.

Analytics (Optional)

If you do not want to report analytics data to the Homebrew project, you can disable a couple of different ways, either via a variable in your shell:

export HOMEBREW_NO_ANALYTICS=1

Or by running the command:

brew analytics off

More info can be found on the Homebrew analytics page.

Brew Update

While we literally just installed Brew, it doesn’t hurt to be exposed how to update it before doing other package installations. The following command will update and upgrade brew itself:

brew update && brew upgrade

Barring bad timing (or a delay in running this command), you should see a message “Already up-to-date”.

Python

Now that our Homebrew package manager is in place, time to install Python. We will be using Python via Visual Studio Code later. Thanks to brew this will be fairly simple. A few things to keep in mind however:

• Python itself can be vary version sensitive when developing with it. This is purely just to get a basic setup going on the current version of Python.

• Python itself has its own package manager named pip3. pip3 is specifically used to manage libraries relevant to Python itself, not tooling on your Mac.

The brew command to install Python3 is:

brew install python3

At this time, version 3.11 is installed. Here is the trailing output from the installation if curious:

You can also confirm your Python3 installation (and version) is working by running:

python3 --version

Visual Studio Code

There, like most applications, many ways to install Visual Studio Code. We will continue to use Homebrew in this post.

Cask

Cask is a project/feature, that is part of Homebrew, that allows the managing of larger precompiled binaries, graphical packages, etc. We will use the cask feature to download Visual Studio Code.

The command to run is as follows:

brew install --cask visual-studio-code

If everything is working as planned, you will see output similar to the image below:

From here, we need to do some simple setup in Visual Studio Code.

Configuring Visual Studio Code

The first thing we need to do is set up where we would like our source code to live. In your terminal, navigate to your home directory and create a directory called “Repos” (short for repositories). You can call it whatever you like, but note “code” is actually the command for launching Visual Studio Code so I would advise against it for clarity.

cd ~
mkdir Repos
cd Repos

Now that you are in your Repos directory launch Visual Studio Code from this folder via:

code .

This tells Visual Studio Code to launch from inside the “Repos” directory.

From here you will have a choice to make regarding what files and directories you will trust as seen below:

For now we only want to work from ~/Repos and below, so just click “Yes, I trust the authors” because for now, that’s yourself.

You will be prompted for some basic configuration choices at this point. It’s up to you what you prefer, or you can even ignore these for now by closing the Welcome tab. I chose to leave the Dark Modern color theme and closed the Welcome tab.

Visual Studio Code Python Configuration

The last configuration remaining is setting up Visual Studio Code to work with Python. Visual Studio Code inherently is just a powerful editor that is highly extensible. We are now going to extend it to work with Python3.

Extensions

Microsoft has provided an Extensions (the building block icons on the lower left) marketplace for all kinds of tools, languages, etc. To extend the editor to work with Python, we will install the Python extension seen above. Clicking Install will install the current version, not a pre-release version which is available in the dropdown.

Once clicked, as shown above, the extension will be installed and this Welcome tab will pop-up. Now we can move on to actually creating and running a Python script.

Python

We can create a new Python script a few different ways, but for this tutorial, simply click the “Create Python File”, or if you already closed the Welcome tab, go to File→New File and create a new file named “hello_python.py”. If you chose the “Create Python File” option go ahead and hit command-s and save the file as “hello_python.py” to your Repos directory. The .py extension will be detected by Visual Studio Code and give you the appropriate options via a Python “language server” running behind the scenes.

Additionally you can now see the the environment has been setup for Python, using the Python3 package you installed earlier via Homebrew by change down in the status bar:

For now, let’s start with the classic printing of “Hello World”. Copy the following into your python script in the editor and save the file.

print("Hello world!")

Running Python

Now to run this script, while open, simply press the play button:

The output will appear down in the terminal below:

This output confirms to us:

• Visual Studio Code is configured correctly to use Python.

• The Python in use is what we installed via Homebrew (can see in the path of the python3 executable being in /opt/homebrew/bin).

• That basic Python will execute in this environment.

Debugging

One last thing. In this example we have changed the code to print a variable, x, that has the value of 1. By clicking in the far left of the editor, in the “gutter”, you can click to leave a breakpoint. This in combination with the dropdown option to “Debug Python File”, will allow you to run your script in debug mode, and have its execution stop when that line (line 3 in this case) is reached. When the breakpoint is hit, you can further debug your script and either step or continue its execution.

Below is the Visual Studio Code IDE in debug mode, stopped at the breakpoint on line 3. To the left you can see the current value of x before it is printed to the terminal (by pressing continue or step over in the mini toolbar above the code).

Chrome URLs represent internal endpoints to the Chrome (Chromium) runtime that are available to the end user via the browser itself.

There are multiple helpful Chrome URL endpoints that you might not be aware of outside of possibly chrome://welcome when you first install Chrome.

chrome://welcome

chrome://bookmarks

NOTE: Your default bookmarks file on a Mac is located at:

/Users/<username>/Library/Application Support/Google/Chrome/Default/Bookmarks

Additional Common Endpoints

chrome://extensions

chrome://version

chrome://dino

chrome://gpu

chrome://device-log

Master URL List

chrome://about

Going to chrome://about will provide a list of Chrome URLs that your current version of Chrome supports.

Chromium

To find additional details on what URLs are available, the Chromium project itself is a good start. Two locations for showing these endpoints include:

chromium / chrome / common / webui_url_constants.cc

and

chromium / content / public / common / url_constants.cc