Quickly cloning all your repositories from GitHub

Sometimes, like when setting up a new computer, you want to quickly clone all the GitHub repositories you work with.

Especially when there are many repositories, cloning the repositories one by one can be very annoying.

Like every engineer, we have an urge to automate our way out of a boring task.

Image courtesy of XKCD

To automate this task, we will be using the GitHub CLI and PowerShell.

For the purpose of this post, I will assume you already have configured the CLI properly to access GitHub on your behalf.

Since the GitHub CLI does not support repository management directly, we will use its api command to interact directly with the API.

The GraphQL query

The GitHub API exposes two surfaces: the REST API and the GraphQL API. I’ll use the GraphQL endpoint to query and fetch all and only the data I need.

The query that we want to execute can be expressed as

Give me the name and the SSH url of the first 100 repositories that I own and are not forks, sorted by name.

We can express this query in GraphQL as follows

query myRepos
{
  viewer
  {
    repositories (first: 100, isFork: false, affiliations: OWNER, orderBy: {direction: ASC, field: NAME})
    {
      nodes
      {
        name,
        sshUrl
      }
    }
  }
}

To execute this GraphQL query, we will use the api graphql command.

> gh api graphql -f query='query myRepos { viewer { repositories(first: 100, isFork: false, affiliations: OWNER, orderBy: {direction: ASC, field: NAME}) { nodes { name, sshUrl } } } }'

This command will give us the following result (trunkated for brevity).

{
  "data":{
    "viewer":{
      "repositories":{
        "nodes":[
          {
            "name":".github",
            "sshUrl":"git@github.com:Kralizek/.github.git"
          },
          {
            "name":"angular-mpa",
            "sshUrl":"git@github.com:Kralizek/angular-mpa.git"
          },
          ...
        ]
      }
    }
  }
}

Even if this is quite a simple query, it would be very hard to write it without having a deep knowledge of the schema of the GraphQL endpoint of the GitHub API.

To solve this issue, GitHub offers a convenient explorer that can be used to compose the queries and run directly in the browser window.

As you can see from the screenshot above, you need to authenticate with your own account to be able to perform queries.

Another important thing to keep in mind is that the Explorer is using real live data, so watch out on what queries and commands are you sending!

Processing the result

As we saw, the JSON document we receive contains all the repositories we want to clone.

We can use PowerShell to process the document and extract all the entries.

This is what we will do

• We save the result of the API call in a variable
• We parse its content using the ConvertFrom-Json cmdlet
• We traverse the tree until we reach the nodes array
• We save the items into a new variable

> $response = (gh api graphql -f query='query myRepos { viewer { repositories(first: 100, isFork: false, affiliations: OWNER, orderBy: {direction: ASC, field: NAME}) { nodes { name, sshUrl } } } }')
> $json = $response | ConvertFrom-Json
> $nodes = $json.data.viewer.repositories.nodes

If we inspect the content of the $nodes variable, we can see that the parsing was successful (result truncated for brevity).

> $nodes

name                 sshUrl
----                 ------
.github              git@github.com:Kralizek/.github.git
angular-mpa          git@github.com:Kralizek/angular-mpa.git
AspNetCore.Metrics   git@github.com:Kralizek/AspNetCore.Metrics.git
Assembla.Connector   git@github.com:Kralizek/Assembla.Connector.git
AutoFixtureCourse    git@github.com:Kralizek/AutoFixtureCourse.git
...

Cloning the repositories

Now that we have a list of all repositories with their name and their SSH url, we can proceed to clone all of them.

Also in this case, we will leverage PowerShell to execute a command on each item of the $nodes array.

> $nodes | ForEach-Object { gh repo clone $_.sshUrl $_.name }

The command above will perform a gh repo clone for each entry of the $nodes array effectively cloning each sshUrl into a folder named after the name property.

Depending on how prolific you are, this can take a while. So take your time for a cup of coffee!

Merging the commands

If you want to have a one-liner to copy/paste when needed, you can use this

> ((gh api graphql -f query='query myRepos { viewer { repositories(first: 100, isFork: false, affiliations: OWNER, orderBy: {direction: ASC, field: NAME}) { nodes { name, sshUrl } } } }') | ConvertFrom-Json).data.viewer.repositories.nodes | ForEach-Object { gh repo clone $_.sshUrl $_.name }

The command above is obtained by simply inlining each variable’s value.

Cloning an organization

The same approach can be used to clone the repositories of an organization.

This is the GraphQL query we will use

query myOrgRepos($org: String!)
{
  organization(login: $org)
  {
    repositories (first: 100, orderBy: {direction: ASC, field: NAME})
    {
      nodes
      {
        name,
        sshUrl
      }
    }
  }
}

Unlike in the previous case, here we are using query parameters to specify the name of the organization to clone.

The GitHub CLI offers the possibility to specify fields of the query by using the -F option.

Here is how to fetch all the resositories of the Insight Architectures organization.

gh api graphql -F org='insight-architectures' -f query='query myOrgRepos($org: String!) { organization(login: $org) { repositories (first: 100){ nodes { name, sshUrl } } } }'

Notice how the parameter specified in the query is then valued by the command argument.

Finally, we can initiate the fork of all repositories of the organization by changing the command to accommodate the different query and result format.

((gh api graphql -F org='insight-architectures' -f query='query myOrgRepos($org: String!) { organization(login: $org) { repositories (first: 100){ nodes { name, sshUrl } } } }') | ConvertFrom-Json).data.organization.repositories.nodes | ForEach-Object { gh repo clone $_.sshUrl $_.name }

Recap

In this post we have seen how to leverage GitHub CLI to access the list of repositories and some of their properties and use this data to perform a clone operation on each entry of the array.

Support this blog

If you liked this article, consider supporting this blog by buying me a pizza!