I decided to communicate with NNTP because of the necessity. I could have used my own NNTP class that I created a long time ago, but I decided to use ruby-net-nntp, which can use Gemfile. This library stores articles in the TMail :: Mail class, so bundle install will also download tmail-1.2.7.1.
I summarized the use of net / nttp and the problems that occurred with tmail.
I prepared the following files.
source 'https://rubygems.org'
gem "ruby-net-nntp"
Download the library using bundler
$ bundle install --path lib
$ ls lib/ruby/2.5.0/gems/
log4r-1.1.10/ ruby-net-nntp-1.0.0/ tmail-1.2.7.1/
When I tried to run the code, I immediately ran into an error.
require "net/nntp"When I run, I get the following error:
Reproduce the error from irb
$ irb
irb(main):001:0> require "rubygems"
=> false
irb(main):002:0> require "bundler/setup"
=> true
irb(main):003:0> require 'net/nntp'
Traceback (most recent call last):
14: from /usr/bin/irb:11:in `<main>'
...
2: from .../lib/ruby/2.5.0/gems/tmail-1.2.7.1/lib/tmail/utils.rb:110:in `<module:TMail>'
1: from .../lib/ruby/2.5.0/gems/tmail-1.2.7.1/lib/tmail/utils.rb:117:in `<module:TextUtils>'
RegexpError (/.../n has a non escaped non ASCII character in non ASCII-8BIT script)
This is [http://www.ownway.info/Blog/2011/09/-invalid-multibyte-escape.html --How to deal with the problem that invalid multibyte escape error occurs in regular expressions](http://www. It is explained in ownway.info/Blog/2011/09/-invalid-multibyte-escape.html).
The link to Google in this article doesn't work, so [https://www.ruby-forum.com/t/ruby-1-9-1-invalid-multibyte-escape-regexperror/164732](https: / You can see the original discussion at /www.ruby-forum.com/t/ruby-1-9-1-invalid-multibyte-escape-regexperror/164732).
Replace the /../n notation where the problem simply occurs with Regexp.new ().
tmail library diff
diff -ru tmail-1.2.7.1.orig/./lib/tmail/scanner_r.rb tmail-1.2.7.1/./lib/tmail/scanner_r.rb
--- tmail-1.2.7.1.orig/./lib/tmail/scanner_r.rb 2020-06-10 22:06:16.813972850 +0900
+++ tmail-1.2.7.1/./lib/tmail/scanner_r.rb 2020-06-11 11:17:51.741512220 +0900
@@ -63,24 +63,24 @@
PATTERN_TABLE = {}
PATTERN_TABLE['EUC'] =
[
- /\A(?:[#{atomchars}]+|#{iso2022str}|#{eucstr})+/n,
- /\A(?:[#{tokenchars}]+|#{iso2022str}|#{eucstr})+/n,
+ Regexp.new("\A(?:[#{atomchars}]+|#{iso2022str}|#{eucstr})+", nil, "n"),
+ Regexp.new("\A(?:[#{tokenchars}]+|#{iso2022str}|#{eucstr})+", nil, "n"),
quoted_with_iso2022,
domlit_with_iso2022,
comment_with_iso2022
]
PATTERN_TABLE['SJIS'] =
[
- /\A(?:[#{atomchars}]+|#{iso2022str}|#{sjisstr})+/n,
- /\A(?:[#{tokenchars}]+|#{iso2022str}|#{sjisstr})+/n,
+ Regexp.new("\A(?:[#{atomchars}]+|#{iso2022str}|#{sjisstr})+", nil, "n"),
+ Regexp.new("\A(?:[#{tokenchars}]+|#{iso2022str}|#{sjisstr})+", nil, "n"),
quoted_with_iso2022,
domlit_with_iso2022,
comment_with_iso2022
]
PATTERN_TABLE['UTF8'] =
[
- /\A(?:[#{atomchars}]+|#{utf8str})+/n,
- /\A(?:[#{tokenchars}]+|#{utf8str})+/n,
+ Regexp.new("\A(?:[#{atomchars}]+|#{utf8str})+", nil, "n"),
+ Regexp.new("\A(?:[#{tokenchars}]+|#{utf8str})+", nil, "n"),
quoted_without_iso2022,
domlit_without_iso2022,
comment_without_iso2022
@@ -258,4 +258,4 @@
end
end # module TMail
-#:startdoc:
\ No newline at end of file
+#:startdoc:
diff -ru tmail-1.2.7.1.orig/./lib/tmail/utils.rb tmail-1.2.7.1/./lib/tmail/utils.rb
--- tmail-1.2.7.1.orig/./lib/tmail/utils.rb 2020-06-10 22:06:16.813972850 +0900
+++ tmail-1.2.7.1/./lib/tmail/utils.rb 2020-06-11 11:11:59.582616384 +0900
@@ -114,10 +114,11 @@
lwsp = %Q| \t\r\n|
control = %Q|\x00-\x1f\x7f-\xff|
- CONTROL_CHAR = /[#{control}]/n
- ATOM_UNSAFE = /[#{Regexp.quote aspecial}#{control}#{lwsp}]/n
- PHRASE_UNSAFE = /[#{Regexp.quote aspecial}#{control}]/n
- TOKEN_UNSAFE = /[#{Regexp.quote tspecial}#{control}#{lwsp}]/n
+ ## reference: http://www.ownway.info/Blog/2011/09/-invalid-multibyte-escape.html
+ CONTROL_CHAR = Regexp.new("[#{control}]", nil, "n")
+ ATOM_UNSAFE = Regexp.new("[#{Regexp.quote aspecial}#{control}#{lwsp}]", nil, "n")
+ PHRASE_UNSAFE = Regexp.new("[#{Regexp.quote aspecial}#{control}]", nil, "n")
+ TOKEN_UNSAFE = Regexp.new("[#{Regexp.quote tspecial}#{control}#{lwsp}]", nil, "n")
# Returns true if the string supplied is free from characters not allowed as an ATOM
def atom_safe?( str )
diff -ru tmail-1.2.7.1.orig/./lib/tmail/vendor/rchardet-1.3/lib/rchardet/universaldetector.rb tmail-1.2.7.1/./lib/tmail/vendor/rchardet-1.3/lib/rchardet/universaldetector.rb
--- tmail-1.2.7.1.orig/./lib/tmail/vendor/rchardet-1.3/lib/rchardet/universaldetector.rb 2020-06-10 22:06:16.829961402 +0900
+++ tmail-1.2.7.1/./lib/tmail/vendor/rchardet-1.3/lib/rchardet/universaldetector.rb 2020-06-11 11:18:58.457912881 +0900
@@ -36,7 +36,7 @@
class UniversalDetector
attr_accessor :result
def initialize
- @_highBitDetector = /[\x80-\xFF]/
+ @_highBitDetector = Regex.new("[\x80-\xFF]")
@_escDetector = /(\033|\~\{)/
@_mEscCharSetProber = nil
@_mCharSetProbers = []
I don't know if there is an open access NetNews server, but the code to access my INN 2.5.5 looks like this:
test.rb
#!/usr/bin/ruby
require "rubygems"
require "bundler/setup"
require 'net/nntp'
nntp = Net::NNTP.new
Net::NNTP.logger = Log4r::Logger.new("nntp")
nntp.host = "nntp.example.com"
nntp.port = 119
welcome = nntp.connect
if Net::NNTP::OKResponse === welcome
group_name = "misc.test"
group_response = nntp.process(Net::NNTP::Group.new(group_name))
listgroup_response = nntp.process(Net::NNTP::Listgroup.new(group_name))
listgroup_response.list.each { |article_id|
}
end
This has nothing to do with Docker, so I'll add it to the section above. For the header of the stored message, some fields have a unique class, but those that do not perform specific processing such as Subject: have their values stored in the ** UnstructuredHeader ** class.
In this, Decoder.decode is called, but this implementation makes NKF.nkf called with the "-mSj" option (MIME decode + Shift-JIS input encoding + ISO-2022-JP output encoding). ..
Since there is a problem if the implementation of Decoder.decode is corrected, the omitted argument is used and the output encoding is set to UTF8.
Also, From: uses the ** AddressHeader ** class that inherits from ** StructuredHeader **, so another action is required. Since there is a check routine that assumes only "jes" (ISO-2022-JP, EUC-JP, Shift-JIS) as the encoding before calling nkf, you can also define the Decoder class a little on this point. Is put in.
Additional diffs to tmail
diff -ur lib/ruby/2.5.0/gems/tmail-1.2.7.1.orig/lib/tmail/header.rb lib/ruby/2.5.0/gems/tmail-1.2.7.1/lib/tmail/header.rb
--- lib/ruby/2.5.0/gems/tmail-1.2.7.1.orig/lib/tmail/header.rb 2020-06-16 10:59:57.589086000 +0900
+++ lib/ruby/2.5.0/gems/tmail-1.2.7.1/lib/tmail/header.rb 2020-06-16 11:01:30.527304391 +0900
@@ -189,5 +189,5 @@
end
def parse
- @body = Decoder.decode(@body.gsub(/\n|\r\n|\r/, ''))
+ @body = Decoder.decode(@body.gsub(/\n|\r\n|\r/, ''), 'w')
end
def isempty?
diff -ur lib/ruby/2.7.0/gems/tmail-1.2.7.1.orig/lib/tmail/encode.rb lib/ruby/2.7.0/gems/tmail-1.2.7.1/lib/tmail/encode.rb
--- lib/ruby/2.7.0/gems/tmail-1.2.7.1.orig/lib/tmail/encode.rb 2020-06-18 14:32:37.352341212 +0900
+++ lib/ruby/2.7.0/gems/tmail-1.2.7.1/lib/tmail/encode.rb 2020-06-18 14:45:57.637057274 +0900
@@ -128,7 +128,7 @@
def initialize( dest, encoding = nil, eol = "\n" )
@f = StrategyInterface.create_dest(dest)
- @encoding = (/\A[ejs]/ === encoding) ? encoding[0,1] : nil
+ @encoding = (/\A[ejsw]/ === encoding) ? encoding[0,1] : nil
@eol = eol
end
diff -ur lib/ruby/2.7.0/gems/tmail-1.2.7.1.orig/lib/tmail/header.rb lib/ruby/2.7.0/gems/tmail-1.2.7.1/lib/tmail/header.rb
--- lib/ruby/2.7.0/gems/tmail-1.2.7.1.orig/lib/tmail/header.rb 2020-06-18 14:32:37.993666420 +0900
+++ lib/ruby/2.7.0/gems/tmail-1.2.7.1/lib/tmail/header.rb 2020-06-18 14:47:16.026796369 +0900
@@ -147,7 +147,7 @@
def body
ensure_parsed
- v = Decoder.new(s = '')
+ v = Decoder.new(s = '', "w")
do_accept v
v.terminate
s
@@ -225,7 +225,7 @@
rescue SyntaxError
if not save and mime_encoded? @body
save = @body
- @body = Decoder.decode(save)
+ @body = Decoder.decode(save, "w")
retry
elsif save
@body = save
When applying a patch in a ruby-2.7 environment such as Docker's ruby: 2.7-alpine image or Ubuntu 20.04 LTS, please convert the file "2.5.0" to "2.7.0" before using it.
2.7 Example of patch application to environment
$ sed -e 's/2.5.0/2.7.0/g' tmail.diff | patch -p0
The Docker image uses ruby: 2.7-alpine. There were some errors, some of which weren't apparent in ruby-2.5, some of which came from trying to run as a non-root user.
The following is a Dockerfile that omits multi stage build etc. for the time being. In the current directory, there is a ruby-sinatra template code created by openapi-generator such as Gemfile and config.ru.
Dockerfile
FROM ruby:2.7-alpine as rubydev
RUN apk --no-cache add tzdata bash ca-certificates make gcc libc-dev linux-headers build-base patch
RUN mkdir /app
COPY . /app
WORKDIR /app
RUN cp /usr/local/include/ruby-2.7.0/ruby/defines.h /usr/local/include/ruby-2.7.0/defines.h
RUN bundle config path lib
RUN bundle install
ENV SINATRA_PORT 8080
EXPOSE $SINATRA_PORT
ADD run.sh /run.sh
RUN chmod +x /run.sh
RUN addgroup sinatra
RUN adduser -S -G sinatra sinatra
# RUN cp -r /root/.bundle /home/sinatra/.bundle
USER sinatra
ENTRYPOINT ["/run.sh"]
An error will occur if defines.h cannot be compiled.
Error encountered
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
...
current directory: /app/lib/ruby/2.7.0/gems/tmail-1.2.7.1/ext/tmailscanner/tmail
make "DESTDIR="
make: *** No rule to make target '/usr/local/include/ruby-2.7.0/defines.h',
needed by 'tmailscanner.o'. Stop.
...
Since defines.h does not exist in the expected location, it was supported by cp.
Workaround added to Dockerfile
RUN cp /usr/local/include/ruby-2.7.0/ruby/defines.h \
/usr/local/include/ruby-2.7.0/defines.h
In Ubuntu 18.04 + ruby-2.5 (deb package) environment, ruby / defines.h was explicitly specified at the end of Makefile, but as far as I checked Makefile of tmail in Dockerfile, -I flag It seems that only /usr/local/include/ruby-2.7.0 is specified in.
ubuntu+ruby-2.5 environments
tmailscanner.o: tmailscanner.c $(hdrdir)/ruby/ruby.h $(arch_hdrdir)/ruby/config.h $(hdrdir)/ruby/defines.h Makefile
In the 2.7-alpine environment, for some reason defines.h should be ruby / defines.h.
2.7-alpine docker image
tmailscanner.o: tmailscanner.c $(hdrdir)/ruby.h $(arch_hdrdir)/ruby/config.h $(hdrdir)/defines.h Makefile
The problem is mkmf.rb, in the pre-processing to write out the depend file, the process to modify the line that is $ (hdrdir) /defines.h to $ (hdrdir) ruby / defines.h is ruby-2.7 mkmf. It was because it was deleted in rb.
mkmf.rb
## ruby-2.5 mkmf.rb
depend.each_line do |line|
line.gsub!(/\.o\b/, ".#{$OBJEXT}")
line.gsub!(/\{\$\(VPATH\)\}/, "") unless $nmake
line.gsub!(/\$\((?:hdr|top)dir\)\/config.h/, $config_h)
line.gsub!(%r"\$\(hdrdir\)/(?!ruby(?![^:;/\s]))(?=[-\w]+\.h)", '\&ruby/')
if $nmake && /\A\s*\$\(RM|COPY\)/ =~ line
## ruby-2.7 mkmf.rb
depend.each_line do |line|
line.gsub!(/\.o\b/, ".#{$OBJEXT}")
line.gsub!(/\{\$\(VPATH\)\}/, "") unless $nmake
line.gsub!(/\$\((?:hdr|top)dir\)\/config.h/, $config_h)
if $nmake && /\A\s*\$\(RM|COPY\)/
It looks like the change committed at https://github.com/ruby/ruby/commit/3d1c86a26f0c96a9c1d0247b968aa96e6f3c30bb#diff-7aa560cb6196deeb96779cd175e8e589.
It seems that I wanted to fix the location of ruby.h, but probably instead of deleting the corresponding line, I added a gsub! Line below this that rewrites "ruby / ruby.h" to "ruby.h" I think I should have done it.
It is registered as https://bugs.ruby-lang.org/issues/16490 in Ruby's Issue Tracking System, but it seems that consideration is not progressing.
When using it with Docker, there is no other way but to take a corresponding workaround by changing the location of defines.h rather than patching mkmf.rb ...
In ruby-2.5, I used $ bundle install --path lib, but I got a warning saying deprecated, so I changed it to use `` `$ bundle config path lib``` doing.
Because of this, it depends on the contents of the ~ / .bundle / config file, so I changed the runtime user with USER sinatra, so I encountered an error because the ~ / .bundle / config file does not exist. did.
Details of the error
$ make docker-run
sudo docker run -it --rm -p 8080:8080 --name nntp-reader nntp-reader:latest
bundler: command not found: rackup
Install missing gem executables with `bundle install`
Makefile:43: recipe for target 'docker-run' failed
make: *** [docker-run] Error 127
The following measures can be considered first.
Workaround 1-run.Resolve at runtime with sh
#!/bin/bash
bundle config path lib ##Added workaround
export SINATRA_PORT=${SINATRA_PORT:-8080}
bundle exec rackup --host 0.0.0.0 --port $SINATRA_PORT
The following method was adopted this time.
Workaround 2-Resolve when creating an image with Dockerfile
RUN cp -r /root/.bundle /home/sinatra/.bundle
that's all
Recommended Posts