Post

Converting ASCII encoded file to UTF-8

You get a file whose encoding you don’t know and want to convert it to UTF-8 encoded file using java. How to do it? Below should work -

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import org.apache.commons.io.IOUtils;
import org.mozilla.universalchardet.UniversalDetector;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.Reader;
public class Test {
public static void main(String[] args) throws IOException {
byte[] buf = new byte[4096];
String fileName = "Test.txt";
java.io.FileInputStream fis = new java.io.FileInputStream(fileName);
UniversalDetector detector = new UniversalDetector(null);
int nread;
while ((nread = fis.read(buf)) > 0 && !detector.isDone()) {
detector.handleData(buf, 0, nread);
}
detector.dataEnd();
String encoding = detector.getDetectedCharset();
detector.reset();
Reader reader = new InputStreamReader(new FileInputStream("Test.txt"), encoding);
byte[] bytes = IOUtils.toString(reader).getBytes("UTF-8");
System.out.println(new String(bytes, "UTF-8"));
}
}

This does rely on another package which is used to detect encoding of file on the fly. This is optional if you already know the source encoding.

This post is licensed under CC BY 4.0 by the author.