Skip to content

複合動詞のフィルターの緩和 #141

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
morning85 wants to merge 2 commits into
base: master
Choose a base branch
from
Open

複合動詞のフィルターの緩和 #141

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
daf9979
addd substring matching to filter function
morning85 May 8, 2025
b9bae42
modified to lower the score of filtered words
morning85 May 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions src/Parser/Language/Japanese/Filter/KWJAFilter.hs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,9 @@ import qualified Text.Juman as J
import qualified Parser.Language.Japanese.Filter.KNPFilter as KF
import qualified Parser.Language.Japanese.Filter.LightblueFilter as LF (getDaihyo, getFeatures)

import qualified Debug.Trace as D
import Text.Show.Unicode (ushow)

type ConjMap = M.Map (T.Text, T.Text) T.Text
type OpenWordsMap = M.Map T.Text [(T.Text, S.Set T.Text)]

Expand Down Expand Up @@ -80,7 +83,7 @@ createFilterFrom ::
createFilterFrom kwjaDatas conjMap _ _ ccgNodes =
-- knpDataから、名詞、動詞、副詞、助詞の表層形を取得
let openWords = getOpenWordsMap kwjaDatas conjMap
in createFilterFrom' openWords ccgNodes
in D.trace (ushow openWords) createFilterFrom' openWords ccgNodes

-- [("名詞"歌"),("動詞","歌")] -> toFilterNode -> FilteredNode
createFilterFrom' :: OpenWordsMap -> [CCG.Node] -> [CCG.Node]
Expand Down Expand Up @@ -129,8 +132,12 @@ filterNodes (key, _) owmap node nodes = case M.lookup key owmap of
Nothing -> let text = CCG.pf node
in LB.verblex [(CCG.pf node)] "(KWJA)" f1' f2' (LF.getDaihyo text) (TL.fromStrict newCaseFrame) LB.event
in newNode ++ createFilterFrom' owmap nodes
-- 表層形が部分一致する場合は残す
| isPartialMatch (CCG.pf node) xs -> node : createFilterFrom' owmap nodes
-- 表層形も格フレームも一致しない場合
| otherwise -> createFilterFrom' owmap nodes
| otherwise ->
let newNode = node{CCG.score = CCG.score node - 20}
in newNode : createFilterFrom' owmap nodes
-- キーが存在しない場合
Nothing -> node : createFilterFrom' owmap nodes
where
Expand All @@ -142,6 +149,11 @@ filterNodes (key, _) owmap node nodes = case M.lookup key owmap of
if pf == word
then T.concat $ S.toList caseframe
else getCaseframe pf xs
-- | 文字列がリスト内のいずれかの単語と部分一致するかを判定
isPartialMatch :: TL.Text -> [(T.Text, S.Set T.Text)] -> Bool
isPartialMatch nodePf openWordList =
any (\(word, _) -> TL.isInfixOf (TL.fromStrict word) nodePf ||
TL.isInfixOf nodePf (TL.fromStrict word)) openWordList

--CCGが名詞どうかの判定
isNoun :: CCG.Cat -> Bool
Expand Down