Mercurial > hg > monetdb-java

annotate src/main/java/org/monetdb/util/FileTransferHandler.java @ 973:32f246853ec4 default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Optimisation, call connection.mapClobAsVarChar() and connection.mapBlobAsVarBinary() outside the for-loop, as it does not change.
author Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
date Thu, 10 Apr 2025 19:26:59 +0200 (2 weeks ago)
parents d416e9b6b3d0
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
542
d462000fc410 Various changes suggested by Martin van Dinther
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 541
diff changeset
1 /*
833
e890195256ac Update copyright for the new year, move to MonetDB Foundation, add SPDX.
Sjoerd Mullender <sjoerd@acm.org>
parents: 716
diff changeset
2 * SPDX-License-Identifier: MPL-2.0
e890195256ac Update copyright for the new year, move to MonetDB Foundation, add SPDX.
Sjoerd Mullender <sjoerd@acm.org>
parents: 716
diff changeset
3 *
542
d462000fc410 Various changes suggested by Martin van Dinther
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 541
diff changeset
4 * This Source Code Form is subject to the terms of the Mozilla Public
d462000fc410 Various changes suggested by Martin van Dinther
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 541
diff changeset
5 * License, v. 2.0. If a copy of the MPL was not distributed with this
d462000fc410 Various changes suggested by Martin van Dinther
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 541
diff changeset
6 * file, You can obtain one at http://mozilla.org/MPL/2.0/.
d462000fc410 Various changes suggested by Martin van Dinther
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 541
diff changeset
7 *
937
d416e9b6b3d0 Update Copyright year.
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 833
diff changeset
8 * Copyright 2024, 2025 MonetDB Foundation;
833
e890195256ac Update copyright for the new year, move to MonetDB Foundation, add SPDX.
Sjoerd Mullender <sjoerd@acm.org>
parents: 716
diff changeset
9 * Copyright August 2008 - 2023 MonetDB B.V.;
e890195256ac Update copyright for the new year, move to MonetDB Foundation, add SPDX.
Sjoerd Mullender <sjoerd@acm.org>
parents: 716
diff changeset
10 * Copyright 1997 - July 2008 CWI.
542
d462000fc410 Various changes suggested by Martin van Dinther
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 541
diff changeset
11 */
d462000fc410 Various changes suggested by Martin van Dinther
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 541
diff changeset
12
504
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
13 package org.monetdb.util;
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
14
514
443780d71bae More renaming. user registers MonetUploadHandler which is handed an Upload
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 504
diff changeset
15 import org.monetdb.jdbc.MonetConnection;
504
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
16
578
687034945b3f Pass expected encoding as a constructor parameter to FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 576
diff changeset
17 import java.io.BufferedReader;
579
72f4437de9be Reencode downloads if necessary
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 578
diff changeset
18 import java.io.BufferedWriter;
504
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
19 import java.io.IOException;
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
20 import java.io.InputStream;
603
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
21 import java.io.InputStreamReader;
579
72f4437de9be Reencode downloads if necessary
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 578
diff changeset
22 import java.io.OutputStream;
603
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
23 import java.io.OutputStreamWriter;
504
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
24 import java.nio.charset.Charset;
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
25 import java.nio.charset.StandardCharsets;
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
26 import java.nio.file.FileSystems;
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
27 import java.nio.file.Files;
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
28 import java.nio.file.Path;
514
443780d71bae More renaming. user registers MonetUploadHandler which is handed an Upload
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 504
diff changeset
29 import java.nio.file.StandardOpenOption;
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
30 import java.util.zip.GZIPInputStream;
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
31 import java.util.zip.GZIPOutputStream;
504
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
32
535
c9d88af06d35 Javadoc and some minor changes
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 515
diff changeset
33 /**
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
34 * Default implementation of UploadHandler and DownloadHandler interfaces
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
35 * for reading from and writing to files on the local file system.
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
36 * It enables support for:
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
37 * COPY .. INTO table FROM 'file-name' ON CLIENT ...
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
38 * and
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
39 * COPY SELECT_query INTO 'file-name' ON CLIENT ...
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
40 * handling.
535
c9d88af06d35 Javadoc and some minor changes
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 515
diff changeset
41 *
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
42 * Currently only file compression format .gz is supported. This is intentionally
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
43 * as other compression formats would introduce dependencies on external
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
44 * libraries which complicates usage of JDBC driver or JdbcClient application.
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
45 * Developers can of course build their own MyFileTransferHandler class
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
46 * and use it instead of this default implementation.
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
47 *
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
48 * A FileTransferHandler object needs to be registered via
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
49 * {@link MonetConnection#setUploadHandler(MonetConnection.UploadHandler)} and/or
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
50 * {@link MonetConnection#setDownloadHandler(MonetConnection.DownloadHandler)}.
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
51 *
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
52 * @author Joeri van Ruth
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
53 * @author Martin van Dinther
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
54 * @version 1.1
535
c9d88af06d35 Javadoc and some minor changes
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 515
diff changeset
55 */
541
31df6a12fd41 Make the MonetUploadHandler and MonetDownloadHandler interfaces part of MonetConnection
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 535
diff changeset
56 public class FileTransferHandler implements MonetConnection.UploadHandler, MonetConnection.DownloadHandler {
504
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
57 private final Path root;
578
687034945b3f Pass expected encoding as a constructor parameter to FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 576
diff changeset
58 private final Charset encoding;
504
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
59
535
c9d88af06d35 Javadoc and some minor changes
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 515
diff changeset
60 /**
c9d88af06d35 Javadoc and some minor changes
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 515
diff changeset
61 * Create a new FileTransferHandler which serves the given directory.
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
62 *
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
63 * @param dir directory Path to read and write files from
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
64 * @param encoding the specified characterSet encoding is used for all data files in the directory
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
65 * when null the Charset.defaultCharset() is used.
535
c9d88af06d35 Javadoc and some minor changes
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 515
diff changeset
66 */
578
687034945b3f Pass expected encoding as a constructor parameter to FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 576
diff changeset
67 public FileTransferHandler(final Path dir, final Charset encoding) {
687034945b3f Pass expected encoding as a constructor parameter to FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 576
diff changeset
68 this.root = dir.toAbsolutePath().normalize();
581
5aef0ea654b1 Take null to mean the default charset
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 579
diff changeset
69 this.encoding = encoding != null ? encoding: Charset.defaultCharset();
504
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
70 }
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
71
535
c9d88af06d35 Javadoc and some minor changes
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 515
diff changeset
72 /**
c9d88af06d35 Javadoc and some minor changes
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 515
diff changeset
73 * Create a new FileTransferHandler which serves the given directory.
c9d88af06d35 Javadoc and some minor changes
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 515
diff changeset
74 *
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
75 * @param dir directory String to read and write files from
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
76 * @param encoding the specified characterSet encoding is used for all data files in the directory
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
77 * when null the Charset.defaultCharset() is used.
535
c9d88af06d35 Javadoc and some minor changes
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 515
diff changeset
78 */
578
687034945b3f Pass expected encoding as a constructor parameter to FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 576
diff changeset
79 public FileTransferHandler(final String dir, final Charset encoding) {
687034945b3f Pass expected encoding as a constructor parameter to FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 576
diff changeset
80 this(FileSystems.getDefault().getPath(dir), encoding);
504
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
81 }
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
82
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
83 /**
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
84 * Read the data from the specified file (in the root directory) and upload it to the server.
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
85 */
576
095e896f9d7a Updated comments. Improved code. Added final keywords
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 571
diff changeset
86 public void handleUpload(final MonetConnection.Upload handle, final String name, final boolean textMode, final long linesToSkip) throws IOException {
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
87 if (name == null || name.isEmpty()) {
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
88 handle.sendError("Missing file name");
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
89 return;
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
90 }
576
095e896f9d7a Updated comments. Improved code. Added final keywords
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 571
diff changeset
91 final Path path = root.resolve(name).normalize();
504
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
92 if (!path.startsWith(root)) {
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
93 handle.sendError("File is not in upload directory: " + root.toString());
504
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
94 return;
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
95 }
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
96 if (!Files.isReadable(path)) {
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
97 handle.sendError("Cannot read file " + path.toString());
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
98 return;
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
99 }
603
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
100
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
101 // In this implementation we ONLY support gzip compression format and none of the other compression formats.
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
102 if (name.endsWith(".bz2") || name.endsWith(".lz4") || name.endsWith(".xz") || name.endsWith(".zip")) {
597
f30b319d57b8 Removed strange character lastIndexOf???('.') in method name. It reported a compilation error on Windows but not on Linux.
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 596
diff changeset
103 final String extension = name.substring(name.lastIndexOf('.'));
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
104 handle.sendError("Specified file compression format " + extension + " is not supported. Only .gz is supported.");
504
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
105 return;
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
106 }
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
107
603
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
108 InputStream byteStream = Files.newInputStream(path);
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
109 if (name.endsWith(".gz")) {
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
110 byteStream = new GZIPInputStream(byteStream, 128 * 1024);
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
111 }
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
112
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
113 if (!textMode || (linesToSkip == 0 && utf8Encoded())) {
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
114 // when !textMode we must upload as a byte stream
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
115 // when utf8Encoded and linesToSkip is 0 it is more efficient to upload as a byte stream
603
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
116 handle.uploadFrom(byteStream);
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
117 byteStream.close();
579
72f4437de9be Reencode downloads if necessary
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 578
diff changeset
118 } else {
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
119 // cannot upload as a byte stream, must deal with encoding and/or linesToSkip
603
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
120 final BufferedReader reader = new BufferedReader(new InputStreamReader(byteStream, encoding));
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
121 handle.uploadFrom(reader, linesToSkip);
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
122 reader.close();
504
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
123 }
8aa70bd8d21f Register upload handler and download handler separately
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents:
diff changeset
124 }
515
7bbcff2c775b Also implement downloadTo(), and use it in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 514
diff changeset
125
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
126 /**
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
127 * Download the data from the server and write it to a new created file in the root directory.
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
128 * When a file with the same name already exists the download request will send an error and NOT overwrite the existing file.
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
129 */
576
095e896f9d7a Updated comments. Improved code. Added final keywords
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 571
diff changeset
130 public void handleDownload(final MonetConnection.Download handle, final String name, final boolean textMode) throws IOException {
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
131 if (name == null || name.isEmpty()) {
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
132 handle.sendError("Missing file name");
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
133 return;
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
134 }
576
095e896f9d7a Updated comments. Improved code. Added final keywords
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 571
diff changeset
135 final Path path = root.resolve(name).normalize();
515
7bbcff2c775b Also implement downloadTo(), and use it in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 514
diff changeset
136 if (!path.startsWith(root)) {
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
137 handle.sendError("File is not in download directory: " + root.toString());
515
7bbcff2c775b Also implement downloadTo(), and use it in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 514
diff changeset
138 return;
7bbcff2c775b Also implement downloadTo(), and use it in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 514
diff changeset
139 }
567
e068d2039e14 Silly bug in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 551
diff changeset
140 if (Files.exists(path)) {
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
141 handle.sendError("File already exists: " + path.toString());
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
142 return;
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
143 }
603
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
144
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
145 // In this implementation we ONLY support gzip compression format and none of the other compression formats.
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
146 if (name.endsWith(".bz2") || name.endsWith(".lz4") || name.endsWith(".xz") || name.endsWith(".zip")) {
597
f30b319d57b8 Removed strange character lastIndexOf???('.') in method name. It reported a compilation error on Windows but not on Linux.
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 596
diff changeset
147 final String extension = name.substring(name.lastIndexOf('.'));
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
148 handle.sendError("Requested file compression format " + extension + " is not supported. Use .gz instead.");
515
7bbcff2c775b Also implement downloadTo(), and use it in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 514
diff changeset
149 return;
7bbcff2c775b Also implement downloadTo(), and use it in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 514
diff changeset
150 }
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
151
603
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
152 OutputStream byteStream = Files.newOutputStream(path, StandardOpenOption.CREATE_NEW);
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
153 if (name.endsWith(".gz")) {
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
154 byteStream = new GZIPOutputStream(byteStream, 128 * 1024);
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
155 }
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
156
595
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
157 if (!textMode || utf8Encoded()) {
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
158 // when !textMode we must download as a byte stream
36df3c89845d Extended FileTransferHandler utility class by adding support for reading from and writing to gzip compressed files when using file name extension .gz
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 581
diff changeset
159 // when utf8Encoded it is more efficient to download as a byte stream
603
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
160 handle.downloadTo(byteStream);
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
161 byteStream.close();
579
72f4437de9be Reencode downloads if necessary
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 578
diff changeset
162 } else {
72f4437de9be Reencode downloads if necessary
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 578
diff changeset
163 // cannot download as a byte stream, must deal with encoding
603
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
164 final BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(byteStream, encoding));
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
165 handle.downloadTo(writer);
7a6b003d65d6 Fix Gzip Reader in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 597
diff changeset
166 writer.close();
579
72f4437de9be Reencode downloads if necessary
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 578
diff changeset
167 }
515
7bbcff2c775b Also implement downloadTo(), and use it in FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 514
diff changeset
168 }
578
687034945b3f Pass expected encoding as a constructor parameter to FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 576
diff changeset
169
579
72f4437de9be Reencode downloads if necessary
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 578
diff changeset
170 public boolean utf8Encoded() {
578
687034945b3f Pass expected encoding as a constructor parameter to FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 576
diff changeset
171 return encoding.equals(StandardCharsets.UTF_8);
687034945b3f Pass expected encoding as a constructor parameter to FileTransferHandler
Joeri van Ruth <joeri.van.ruth@monetdbsolutions.com>
parents: 576
diff changeset
172 }
576
095e896f9d7a Updated comments. Improved code. Added final keywords
Martin van Dinther <martin.van.dinther@monetdbsolutions.com>
parents: 571
diff changeset
173 }