Truncated chunk

[domlatham]

We are regularly unable to access files with the following error:

Cannot open the specified file.
Truncated chunk ( expected size: 31868; actual size: 22074)

(numbers vary)

Under Details we get:

org.apache.http.TruncatedChunkException: Truncated chunk ( expected size: 31868; actual size: 22074)
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:200)
at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:238)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117)
at org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:73)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at ro.sync.io.f.read(Unknown Source)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at ro.sync.basic.xml.encoding.EncodingDetectorImpl$EncodingErrorReportingReader.read(EncodingDetectorImpl.java:1492)
at ro.sync.exml.editor.xmleditor.prettyprint.c$1.read(Unknown Source)
at ro.sync.exml.editor.y.read(Unknown Source)
at java.io.BufferedReader.read1(BufferedReader.java:210)
at java.io.BufferedReader.read(BufferedReader.java:286)
at ro.sync.basic.io.RewindableReader.read(RewindableReader.java:230)
at ro.sync.basic.io.EmptyLineBufferedReader.fill(EmptyLineBufferedReader.java:119)
at ro.sync.basic.io.EmptyLineBufferedReader.readLine(EmptyLineBufferedReader.java:303)
at ro.sync.basic.io.EmptyLineBufferedReader.readLine(EmptyLineBufferedReader.java:373)
at ro.sync.basic.io.IOUtil.readWithLongLineDetection(IOUtil.java:615)
at ro.sync.basic.io.IOUtil.readWithLongLineDetection(IOUtil.java:582)
at ro.sync.basic.io.IOUtil.readWithLongLineDetection(IOUtil.java:569)
at ro.sync.exml.editor.pagetext.f.ibm(Unknown Source)
at ro.sync.exml.editor.pagetext.f.loadPageData(Unknown Source)
at ro.sync.exml.editor.q.loadPage(Unknown Source)
at ro.sync.exml.editor.xmleditor.e.j.loadPage(Unknown Source)
at ro.sync.exml.editor.xmleditor.e.c.loadPage(Unknown Source)
at ro.sync.exml.editor.wc.readDocument(Unknown Source)
at ro.sync.exml.editor.validate.r.readDocument(Unknown Source)
at ro.sync.exml.editor.xmleditor.rb.readDocument(Unknown Source)
at ro.sync.exml.editor.me.y(Unknown Source)
at ro.sync.exml.editor.me.hb(Unknown Source)
at ro.sync.exml.editor.wc.open(Unknown Source)
at ro.sync.exml.editor.xmleditor.rb.open(Unknown Source)
at ro.sync.exml.editor.ae.ati(Unknown Source)
at ro.sync.exml.editor.ae.open(Unknown Source)
at ro.sync.exml.k.v.c(Unknown Source)
at ro.sync.exml.k.v.d(Unknown Source)
at ro.sync.exml.k.v.e(Unknown Source)
at ro.sync.exml.editor.s.bb$36.b(Unknown Source)
at ro.sync.db.nxd.c.jk(Unknown Source)
at ro.sync.db.nxd.c$4.bli(Unknown Source)
at ro.sync.db.f.z.byp(Unknown Source)
at ro.sync.db.f.z$1.dxd(Unknown Source)
at ro.sync.ui.application.mb.run(Unknown Source)

We are using Oxygen 21 connecting to eXist 4.6.1 using WebDAV.
I appreciate that is a lot of links in the chain but as the error is being raised by Oxygen I was hoping you could help shed some light on what is going on.

With many thanks

[Radu]

Hi,

Oxygen and the Exist server are communicating using HTTP. Sometimes when using HTTP the server may indicate that it can send the content as zipped in gzip format (as it minimizes the bandwidth). This is the case here, Oxygen receives zipped content from the server, its HTTP libraries try to unzip it and fail to do that.

Did you obtain a similar error with previous Exist server versions?
Do you have an HTTP proxy web server set up in Oxygen (Preferences->Network Connection Settings / Proxy)? If so can you try to connect to the server bypassing the HTTP proxy?

Regards,
Radu

[domlatham]

Dear Radu,

Thanks for looking into this.

Did you obtain a similar error with previous Exist server versions?

We had no problem with our previous version of exist 3.4.1. The issues started on upgrading to 4.6.1. We have now upgraded to 4.7.0 but the issue persists.

Do you have an HTTP proxy web server set up in Oxygen (Preferences->Network Connection Settings / Proxy)? If so can you try to connect to the server bypassing the HTTP proxy?

No. It is set to "Use systems settings". I tried "Direct connection" but that didn't help.

With best wishes,

Dominic

[Radu]

Hi Dominic,

I see you already asked on the Exist users list about this? I revived your thread, maybe someone else has seen this problem.
There could be two potential problems here:

1) At some point your network settings changed (maybe some internal routing mechanism started discarded HTTP packages which are over a certain limit). If this would be true then you could also switch to Exist 3.4.1 and the problem could be reproduced even with that server.

2) There is a bug (or a change) which was added in Exist 4.6.1 which makes the problem sometimes appear.

Do you consistently obtain this error for a certain XML file? If so, what exactly do you do when the error is obtained? Double click the XML file in the Data Source Explorer view? Is it a very large XML file? Can you always reproduce the problem for XML files over a certain file size?

Regards,
Radu

[Contra8]

We are facing the same problem here. We have a running system eXist-db 2.2 and Oxygen 20. Works fine.

This week, I installed an eXist 4.6, restored the content and users from 2.2 database. The problem occurs with bigger XML files, starting at a file sizes of around 950 kB.



Detailed error message:

org.apache.http.TruncatedChunkException: Truncated chunk ( expected size: 30726; actual size: 15856)
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:200)
at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
at java.util.zip.InflaterInputStream.fill(Unknown Source)
at java.util.zip.InflaterInputStream.read(Unknown Source)
at java.util.zip.GZIPInputStream.read(Unknown Source)
at org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:73)
at java.io.FilterInputStream.read(Unknown Source)
at ro.sync.io.f.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
at sun.nio.cs.StreamDecoder.read(Unknown Source)
at java.io.InputStreamReader.read(Unknown Source)
at ro.sync.basic.xml.encoding.EncodingDetectorImpl$EncodingErrorReportingReader.read(EncodingDetectorImpl.java:1494)
at ro.sync.exml.editor.xmleditor.prettyprint.c$1.read(Unknown Source)
at ro.sync.exml.editor.x.read(Unknown Source)
at java.io.BufferedReader.read1(Unknown Source)
at java.io.BufferedReader.read(Unknown Source)
at ro.sync.basic.io.RewindableReader.read(RewindableReader.java:230)
at ro.sync.basic.io.EmptyLineBufferedReader.fill(EmptyLineBufferedReader.java:164)
at ro.sync.basic.io.EmptyLineBufferedReader.readLine(EmptyLineBufferedReader.java:348)
at ro.sync.basic.io.EmptyLineBufferedReader.readLine(EmptyLineBufferedReader.java:418)
at ro.sync.basic.io.IOUtil.readWithLongLineDetection(IOUtil.java:660)
at ro.sync.basic.io.IOUtil.readWithLongLineDetection(IOUtil.java:627)
at ro.sync.basic.io.IOUtil.readWithLongLineDetection(IOUtil.java:614)
at ro.sync.exml.editor.pagetext.f.iij(Unknown Source)
at ro.sync.exml.editor.pagetext.f.pvh(Unknown Source)
at ro.sync.exml.editor.p.ndg(Unknown Source)
at ro.sync.exml.editor.xmleditor.e.j.ndg(Unknown Source)
at ro.sync.exml.editor.xmleditor.e.c.ndg(Unknown Source)
at ro.sync.exml.editor.uc.readDocument(Unknown Source)
at ro.sync.exml.editor.validate.q.readDocument(Unknown Source)
at ro.sync.exml.editor.xmleditor.qb.readDocument(Unknown Source)
at ro.sync.exml.editor.ke.x(Unknown Source)
at ro.sync.exml.editor.ke.fb(Unknown Source)
at ro.sync.exml.editor.uc.open(Unknown Source)
at ro.sync.exml.editor.xmleditor.qb.open(Unknown Source)
at ro.sync.exml.editor.yd.sgk(Unknown Source)
at ro.sync.exml.editor.yd.open(Unknown Source)
at ro.sync.exml.k.u.c(Unknown Source)
at ro.sync.exml.k.u.d(Unknown Source)
at ro.sync.exml.k.u.e(Unknown Source)
at ro.sync.exml.editor.s.ab$38.b(Unknown Source)
at ro.sync.db.nxd.c.am(Unknown Source)
at ro.sync.db.nxd.c$4.qqh(Unknown Source)
at ro.sync.db.f.bb.xup(Unknown Source)
at ro.sync.db.f.bb$1.cud(Unknown Source)
at ro.sync.ui.application.nb.run(Unknown Source)

[Radu]

Hi,

Thanks for joining in on this. Does the problem occur consistently? Does it occur consistently for any XML file over 1MB or only for some of them? If it occurs only for some XML files can I perhaps get a sample XML file to test on my side (support@oxygenxml.com)?
What I managed to test on my side were two Exist versions (version 4.6.0 and 4.7.0), in both I uploaded three XML docs, all of them over 1 MB, the largest has 3MBs. But I am able to open the XML files from the server in Oxygen 21.1 and work with them. I am unable to find a place from where to download Exist 4.6.1 so I am unable to test with it.

Regards,
Radu

[Contra8]

This problem occurs permanently for every file over a certain size which is around 1 MB. I will send you an example file to the given e-mail address.

I got eXist 4.6.1 from JFrog: https://bintray.com/existdb/releases/exist/4.6.1/view

Maybe I should give it a try with 4.7.0 and see if it works there ...

[DannesW]

Please use the REST interface instead of the webDAV interface, as the content-length header of GET on this interface might not give the accurate value. We do this for performance optimization, MOST webdav clients can deal with this.

There are a few exist-db settings you can change. These are documented in the eXist-db documentation (sorry don;t have a server available at this moment)

[Radu]

Hi,

So indeed in Oxygen you can either:

1) Connect using Webdav to the Exist server by connecting a Webdav connection in the Preferences Data Sources page.
or
2) You can create an Exist data source using the "Create eXist-db XML connection" link in the same Preferences Data Sources page.

So Dannes' advice is for you to use the (2) approach.
I still cannot reproduce the problem using either the (1) or the (2) approach with Exist 4.7.0 and Oxygen 21.1 to open some 3MB sized files.

Update: tested with Exist 4.6.1 as well and still no luck reproducing the problem on my side using the over 1MB sample XML documents I have.
So having a sample XML document to work with would help.
But you should also try connecting to the Exist server using the (2) approach.

Regards,
Radu

[Radu]

Hi,

Using Dominic's sample I managed to obtain this HTTP exception when opening the file from an Exist 4.6.1 file:

Code: Select all

org.apache.http.MalformedChunkCodingException: CRLF expected at end of chunk
	at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:255)
	at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:227)
	at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:186)
	at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)

I only obtained the error once although I opened the XML document lots of times so strangely the error is not consistent, at least on my side.
I'm adding a link also to the post suggested by Joe Wicentowski as a workaround for the problem: https://github.com/eXist-db/exist/issue ... -435027483

Dominic added the parameter on his server so he'll probably report back if the problem seems to be fixed or not.

Regards,
Radu

[Contra8]

I tried now to use different versions of eXist but the problem stays the same with 4.7, 4.6.1, 3.*.

Dear Radu, I sent you an XML file that is affected via e-mail to support@ixygenxml.com. We created those files when working with eXist 2.2. Now after upgrading we face the problem described here permanently.

[Radu]

Hi,

Can you both confirm that if you uncheck the checbox "Lock WebDav Files on open" from the Preferences->"Network Connection Settings / HTTP(S)/WebDAV" page you no longer can reproduce the problem when opening the large XML documents via WebDav?
If so, I have a couple ideas about what to do in our code in order to work around this problem in a future Oxygen version.

Regards,
Radu

[Contra8]

Dear Radu,

sorry for replying late. I had to check several clients in my team.

You are right, when we uncheck the "Lock WebDAV files on open" in Oxygen preferences the problem disappears. Even the large files are opened immediately. No error message occurs.

I checked this with several Oxygen clients 20.0 on both Windows and Mac.

Interesting side note: I have a team member who runs Oxygen 18.0 on a Mac. On this machine, the described problem does not appear at all. Even when he checks "Lock WebDAV files on open" in Oxygen preferences, the large files open up immediately without any time-out or error message.

Kind regards
Christian

[Radu]

Hi,

We made a minor bug fix release of Oxygen XML 21.1 and it should have the problem with opening large XML documents from Exist fixed:
https://www.oxygenxml.com/build_history.html#2019071807

Regards,
Radu