Dealing With The Unexpected Token in Json Error

If you’ve every created JSON files with PowerShell you may be familiar with the dreaded SyntaxError: Unexpected token in JSON at position 0 error when your JavaScript application attempts to parse the file.

Sometimes this is caused by incorrectly formatted JSON, but if you used ConvertTo-Json, this is unlikely to be the case.

What I’ve found to be the most common culprit is that the file is UTF8 encoded (which is fine), but contains a BOM. The BOM stands for Byte Order Mark and is a sequence of bytes (EF BB BF) at the start of the file that makes it easier to identify the encoding as being UTF8.

Having a BOM isn’t actually required, and JavaScript’s JSON.parse() takes issue with it. However, PowerShell’s Out-File, which you may be using to write the JSON to disk, writes UTF8 files with a BOM.

Now that we know the cause, let’s look at a few solutions.

PowerShell 6

If you’re using PowerShell 6, Out-File provides an encoding option of UTF8NoBOM

Fixing PowerShell < 6 Output

For older versions can use .NET methods to write a UTF8 encoded file with no BOM.

$JSONData = $Data | ConvertTo-Json -Depth 100

# specifying this is optional as utf8-no-bom is the default, but for the sake of being explicit...
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False

[System.IO.File]::WriteAllLines("C:\path\to\file.json", $JSONData, $Utf8NoBomEncoding)

Quick and dirty

If you don’t need UTF8 and are happy for your JSON to be saved as ASCII, you can specify that as an encoding option.

Out-File -Encoding ascii

I don’t have access to the generating code

If you don’t have access to modify how the data is generated you can implement the fix inside the JavaScript app, we simply need to remove the BOM before we call JSON.parse().

Here is a quick node example that reads the file, removes the BOM, and writes a new copy to disk.

const fs = require('fs');

fs.readFile('./data.json', 'utf8', function (err, data) {
    d = data.toString('utf8').replace(/^\uFEFF/, '');
    fs.writeFileSync('./clean_data.json', d, (err) =>{
        if (err) throw err;      
    })
});

Edit: An easier method was mentioned on a stackoverflow post which uses .trim(). Our code then simply becomes:

const fs = require('fs');

fs.readFile('./data.json', 'utf8', function (err, data) {
    fs.writeFileSync('./clean_data.json', data.trim(), (err) =>{
        if (err) throw err;      
    })
});

.trim() is also a great method if we’re getting the JSON data using fetch or axios from an API of some sort.